Information Processing System and Information Processing Method

Abstract

To easily update registered feature point group data associated with position and orientation in a space.SOLUTION: An information processing method includes the steps of: generating, by an information terminal 1, captured image data by imaging at least a partial area of a space to which the information terminal 1 belongs; transmitting feature point group data composed of multiple feature point groups of a subject included in the captured image data generated by the information terminal 1, from the information terminal 1 to an information processing apparatus 2; and causing, by the information processing apparatus 2, a storage apparatus 4 to store the feature point group data in association with positions in the space when a determination is made that the feature point group data received from the information terminal 1 is the latest feature point group data.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an information processing system and an information processing method for displaying an augmented reality image.

Background Art

[0002] Conventionally, there has been known a system that identifies an augmented reality image to be provided to an information terminal based on feature points of a subject included in a captured image generated by the information terminal capturing the subject, and overlays and displays the augmented reality image on the captured image on a display unit (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to identify the position of an information terminal based on feature point group data created by the information terminal identifying a plurality of feature points of a subject, it is necessary to compare the created feature point group data with a large number of registered feature point group data associated in advance with positions and orientations in space. However, when a new building is constructed or a tree grows in space, the registered feature point group data created in the past may not correspond to the state of the latest subject in space. As a result, there has been a problem that the position of the information terminal cannot be identified.

[0005] Therefore, the present invention has been made in view of these points, and an object thereof is to enable easy updating of registered feature point group data associated with positions and orientations in space.

Means for Solving the Problems

[0006] The information processing system according to the first aspect of the present invention includes an information terminal, an information processing device capable of communicating with the information terminal, and a storage device. The information terminal includes an imaging unit that generates imaging image data by imaging at least a part of the space to which the information terminal belongs, and a terminal transmission processing unit that transmits feature point group data composed of a plurality of feature point groups of a subject included in the imaging image data to the information processing device. The information processing device includes a terminal data receiving unit that receives the feature point group data from the information terminal, and a space data management unit that stores the feature point group data received from the information terminal in the storage device in association with a position in the space. When the space data management unit determines that the feature point group data received from the information terminal is the latest feature point group data, the space data management unit stores the feature point group data received from the information terminal in the storage device as registered feature point group data.

[0007] The terminal data receiving unit further acquires position data indicating the position and orientation of the information terminal, and when the feature point group data associated with the position and orientation indicated by the position data is not stored in the storage device, the space data management unit may store the feature point group data received from the information terminal in the storage device as the registered feature point group data.

[0008] When the terminal data receiving unit receives a plurality of the feature point group data from a plurality of the information terminals that have transmitted the position data indicating the same position and orientation, the space data management unit may store the feature point group data with a larger data amount in the storage device as the registered feature point group data.

[0009] When the registered feature point group data corresponding to the position and orientation indicated by the position data is stored in the storage device, the space data management unit may store the feature point group data received from the information terminal in the storage device as the registered feature point group data on the condition that the data amount of the feature point group data received from the information terminal is larger than the data amount of the stored registered feature point group data.

[0010] The spatial data management unit may store the feature point group data in the storage device on the condition that the feature point group data received from the information terminal includes data indicating a subject having a shape registered in advance.

[0011] The information terminal further includes a terminal storage unit that stores a plurality of pieces of the registered feature point group data stored in the storage device. When there is no registered feature point group data in the plurality of registered feature point group data stored in the terminal storage unit whose difference from the feature point group data is equal to or less than a threshold value, the terminal transmission processing unit may transmit the feature point group data to the information processing device.

[0012] An information processing method according to a second aspect of the present invention includes a step of an information terminal generating captured image data by capturing at least a partial area of a space to which the information terminal belongs, a step of transmitting, from the information terminal to an information processing device, feature point group data constituted by a plurality of feature point groups of a subject included in the captured image data generated by the information terminal, and a step of the information processing device storing the feature point group data in a storage device in association with a position in the space when it is determined that the feature point group data received from the information terminal is the latest feature point group data.

Advantages of the Invention

[0013] According to the present invention, there is an effect that the registered feature point group data associated with the position and orientation in the space can be easily updated.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0015] [Outline of Information Processing System S] (System Configuration) FIG. 1 is a diagram showing an outline of an information processing system S. The information processing system S is a system for causing a plurality of information terminals 1 to display an augmented reality image (hereinafter referred to as an “AR image”). The information processing system S includes a plurality of information terminals 1, an information processing apparatus 2, a computer group 3, and a storage device 4.

[0016] The information terminal 1 is a terminal having an imaging function, such as a smartphone, a tablet, or a glasses-type terminal. The user using the information terminal 1 can move to any location. In FIG. 1, a state is shown in which a plurality of information terminals 1a (1a-1 to 1a-m, where m is a natural number) exist in the first space and a plurality of information terminals 1b (1b-1 to 1b-n, where n is a natural number) exist in the second space. The first space and the second space correspond to regions set to display the same AR image for the information terminals 1 of users in the same space, such as one room partitioned by a wall, one floor in a building, one building, or one town. A plurality of information terminals 1 that display the same AR image within the space of the same coordinate system constitute an information terminal group belonging to the space.

[0017] The information processing device 2 is a server that provides various data to the information terminal 1. The information processing device 2 provides, for example, spatial data used by the information terminal 1 to identify its own position and orientation, and AR image data to be displayed on the information terminal 1. The spatial data includes a plurality of pre-created registered feature point group data sets composed of a group of feature points indicating the features of one or more subjects visible when looking at the space at a plurality of positions within the space. In the spatial data, coordinate data indicating the position and orientation when visually recognizing the space is associated with the plurality of registered feature point group data sets. Further, pixel values (R value, G value, B value) indicating color and coordinates indicating positions in the space may be associated with the plurality of feature points included in the registered feature point group data.

[0018] The computer group 3 executes a matching process for identifying the space to which the information terminal 1 belongs. The computer group 3 acquires imaging image data captured and generated by the information terminal 1 or feature point group data indicating the feature points of the subject included in the imaging image data, and performs a matching process on the acquired imaging image data or feature point group data with a plurality of spatial data sets composed of a plurality of registered feature point group data corresponding to each of the plurality of spaces, thereby identifying the space to which the information terminal 1 belongs. Pixel values (R value, G value, B value) indicating color may be associated with the plurality of feature points included in the feature point group data.

[0019] The computer group 3 selects spatial data including a plurality of registered feature point group data corresponding to the space to which the information terminal 1 belongs, and provides the selected spatial data to the information terminal 1. Although details will be described later, the computer group 3 has a plurality of computers 30 that store spatial data corresponding to each of the plurality of spaces, and in order to identify the space to which the information terminal 1 belongs in a short time, the plurality of computers 30 execute matching processes in parallel.

[0020] The storage device 4 is a device that stores spatial data corresponding to various spaces. The storage device 4 stores, for example, spatial data corresponding to a large number of spaces around the world, and among the plurality of spatial data, provides the spatial data corresponding to each of the plurality of computers 30 included in the computer group 3 to each computer 30.

[0021] (Basic operation) FIG. 2 is a diagram for explaining the basic operation of the information processing system S. In FIG. 2, in response to an operation by the user U1 using the information terminal 1a-1 existing in the first space, the same AR image is displayed by being superimposed on an image of the space visually recognized by the user U1 of the information terminal 1a-1 and the user U2 of the information terminal 1a-2 existing in the same first space. Since the information terminal 1a-1 and the information terminal 1a-2 face different directions at different positions in the first space, the same AR image is displayed in different directions at different positions on the screens being displayed by each of them.

[0022] The images of the space visible to users U1 and U2 are, for example, captured images generated by information terminal 1 capturing the space where users U1 and U2 are located. When information terminal 1 is a glasses-type terminal, the images of the space visible to users U1 and U2 may be images that are projected onto the eyes of users U1 and U2 by the light incident on the eyes of users U1 and U2 through the transparent lenses of the glasses-type terminal. In the following description, mainly, the case where the images of the space visible to users U1 and U2 are captured images will be described as an example, but the following configurations and processes are also applicable when the images of the space visible to users U1 and U2 are images projected onto the eyes of users U1 and U2 through lenses.

[0023] Information terminal 1 acquires, from information processing device 2, spatial data indicating the positions of a group of feature points of a subject visible to user U at a plurality of positions within the space where user U is present, and stores the spatial data corresponding to a predetermined space at least while user U is in the predetermined space. The group of feature points is an aggregate of a plurality of points indicating the features of the shape of the subject, and is composed of a plurality of points on the contour line of the subject included in the captured image data generated by photographing the space. In the spatial data, a combination of a position and orientation within a three-dimensional space determined by latitude, longitude, altitude, and azimuth is associated with the group of feature point data.

[0024] FIG. 3 is a diagram for explaining the screens displayed on a plurality of information terminals 1. FIG. 3(a) is a diagram showing the space photographed by information terminals 1a-1 and 1a-2 from above. In FIG. 2(a), the left-right direction is the X direction, the up-down direction is the Y direction, and the depth direction is the Z direction.

[0025] In FIG. 3, it is assumed that the user U1 using the information terminal 1a-1 and the user U2 using the information terminal 1a-2 are at the positions shown in FIG. 3(a). The subjects T1 to T4 in FIG. 3 are subjects existing in the first space to which the information terminal 1a-1 and the information terminal 1a-2 belong. The images A (person), B (bird), and C (arrow) in FIG. 3 are AR images displayed on the information terminal 1a-1 and the information terminal 1a-2.

[0026] FIG. 3(b) shows an example of the screen displayed on the information terminal 1a-1. FIG. 3(b) shows a state where the user U1 has performed an operation of releasing the image C (arrow) from the image A (person) toward the image B (bird) with the hand H of the user U1. FIG. 3(c) shows the image displayed on the information terminal 1a-2 when the operation shown in FIG. 3(b) is performed on the information terminal 1a-1. In FIG. 3(c), the image C is displayed in a different orientation at a position different from that in FIG. 3(b). By the information terminal 1a-1 and the information terminal 1a-2 displaying such images, the user U1 and the user U2 can view the same AR image from different orientations overlaid on the subjects visible at different positions in the same space on the screens of the information terminals 1 they use.

[0027] Hereinafter, with reference to FIG. 2, the flow of the process in which the information terminal 1a-1 and the information terminal 1a-2 display the images shown in FIG. 3 will be described. The information terminal 1 creates feature point group data based on the captured image data generated by the built-in camera by executing application software such as a game or a sightseeing guide that has a function of displaying AR images.

[0028] FIG. 4 is a diagram showing the feature point group data corresponding to the subjects T1 to T4 in the first space shown in FIG. 3. FIG. 4(a) shows the feature point group data of the subjects T1 to T4 photographed by the information terminal 1a-1, and FIG. 4(b) shows the feature point group data of the subjects T1 to T4 photographed by the information terminal 1a-2. Thus, the positions of the plurality of feature points included in the feature point group data are different depending on the position and orientation at which the information terminal 1 photographs.

[0029] The information terminal 1 compares the created feature point group data with a plurality of registered feature point group data included in the pre-stored spatial data, and identifies the registered feature point group data that is most similar to the created feature point group data among the plurality of registered feature point group data. The information terminal 1 identifies the position and orientation of the information terminal 1 by identifying the position and orientation corresponding to the identified registered feature point group data.

[0030] When the information terminal 1a-1 receives a user operation that serves as a trigger for displaying an AR image, as shown in FIG. 2, it notifies the information processing apparatus 2 of operation data indicating the content of the operation and position coordinate data indicating the identified position and orientation. The information terminal 1a-1 acquires an AR image corresponding to the position and orientation from the information processing apparatus 2, and displays the acquired AR image as shown in, for example, FIG. 3(b).

[0031] The information processing apparatus 2 has a plurality of memory databases corresponding to a plurality of spaces. In response to the position coordinate data notified from the information terminal 1a-1 being written into the memory database corresponding to the space to which the information terminal 1a-1 belongs, it transmits an AR image, display coordinate data indicating the position where the AR image is to be displayed, and the direction in which to move, to another information terminal 1a-2 that is an information terminal 1 belonging to the same space. The information terminal 1a-2 displays an AR image corresponding to the position and orientation indicated by the received display coordinate data as shown in, for example, FIG. 3(c).

[0032] In a system for displaying an AR image as in the present embodiment, since a high-speed response is required, it is desirable that the database be constructed on the main memory and be an in-memory database that can be accessed at a higher speed than a database stored in a disk storage mechanism. In the following description, the case where the database is an in-memory database will be exemplified.

[0033] (Provision of Spatial Data) FIG. 5 is a diagram showing an outline of a process in which the information terminal 1 acquires space data corresponding to the space to which the information terminal 1 belongs. The information terminal 1 acquires space data triggered by, for example, starting a predetermined application software, the user U logging in to a predetermined site, or the information terminal 1 moving to a predetermined position.

[0034] When the information terminal 1 detects, for example, that a predetermined application software has been started, the information terminal 1 creates feature point group data based on captured image data generated by capturing an image with a built-in camera. The information terminal 1 transmits start information indicating that the application software has been started and the created feature point group data to the information processing device 2. In response to receiving the start information, the information processing device 2 transmits the received feature point group data to the computer group 3.

[0035] The computer group 3 inputs the received feature point group data to a plurality of computers 30 and executes matching processing in the plurality of computers 30. Among the plurality of computers 30, the computer 30 that detects that the pattern of the feature point group indicated by the received feature point group data matches the pattern of the registered feature point group indicated by the space data stored by itself notifies the information processing device 2 of the match and transmits the space data stored by itself to the information processing device 2. The information processing device 2 transmits the received space data to the information terminal 1.

[0036] The information terminal 1 may transmit, to the information processing device 2 together with the start information, approximate position data indicating an approximate position detected by a GPS receiver. In this case, the information processing device 2 can reduce the number of computers 30 that execute the matching process by transmitting the feature point group data to the computer 30 corresponding to the space including the position indicated by the received approximate position data, so that the load on the computer group 3 is reduced.

[0037] (Update of Space Data) Objects such as buildings, articles, and plants in space change over time. Therefore, the spatial data stored in the memory device 4 does not necessarily correspond to the latest state of the space. Also, there may be a space for which spatial data has not been created. However, it is not easy to collect spatial data every time the state of the space changes. Therefore, the information processing system S updates the spatial data to the latest state using the feature point group data created by a large number of information terminals 1.

[0038] FIG. 6 is a diagram showing an outline of a process for updating spatial data. When the information terminal 1 does not include registered feature point group data that matches the feature point group data created by itself among the plurality of registered feature point group data included in the spatial data acquired via the information processing apparatus 2, it determines that there is an error in the spatial data or that there is no spatial data for the space to which the information terminal 1 belongs. In this case, the information terminal 1 transmits an update request to the information processing apparatus 2 requesting to update the registered feature point group data together with the feature point group data created by itself and position data indicating its own position. The position data is, for example, data indicating latitude, longitude, altitude, and azimuth.

[0039] It is desirable that the position data be as accurate as possible. Therefore, the information terminal 1 calculates the position corresponding to the feature point group data to be transmitted together with the update request based on, for example, the moving distance and moving direction from the position where the registered feature point group data included in the spatial data immediately before and the feature point group data created by itself matched. The information terminal 1 may transmit an update request on the condition that the distance from the position where the registered feature point group data included in the spatial data immediately before and the feature point group data created by itself matched is within a predetermined threshold. The predetermined threshold is determined based on at least either the accuracy of the position indicated by the spatial data or the accuracy with which the information terminal 1 can specify the moving distance and moving direction.

[0040] When the information processing device 2 receives an update request, it transmits the received feature point group data and position data to the storage device 4. When the information processing device 2 receives feature point group data corresponding to the same position and orientation from a plurality of information terminals 1, it transmits the feature point group data that is considered to be more accurate. The feature point group data that is considered to be more accurate is, for example, feature point group data containing a larger number of feature points included in the feature point group data, or feature point group data transmitted from an information terminal 1 with a higher accuracy of the sensor used to identify the position.

[0041] When the storage device 4 receives feature point group data and position data from the information terminal 1, it stores the feature point group data as registered feature point group data included in the spatial data in association with the position and orientation indicated by the received position data (that is, latitude, longitude, altitude, and azimuth). By updating the registered feature point group data included in the spatial data in this way, the information terminal 1 can identify its own position and orientation using the spatial data corresponding to the latest space, and can display the AR image at an appropriate position.

[0042] [Configuration of Each Part] (Configuration of Information Terminal) FIG. 7 is a diagram showing the configuration of the information terminal 1. The information terminal 1 includes a communication unit 11, an operation unit 12, a position detection unit 13, an imaging unit 14, a display unit 15, a storage unit 16, and a control unit 17. The control unit 17 includes a transmission processing unit 171, a reception processing unit 172, an operation reception unit 173, a rough position identification unit 174, an image processing unit 175, a detailed position identification unit 176, a display processing unit 177, and a spatial data acquisition unit 178.

[0043] The communication unit 11 has a communication interface for connecting to a communication network. The communication unit 11 transmits and receives various data to and from the information processing device 2 via a mobile wireless communication network or a wireless communication network such as Wi-Fi (registered trademark). The communication unit 11 transmits the data input from the transmission processing unit 171 and inputs the data received via the communication network to the reception processing unit 172.

[0044] The operation unit 12 has a device for detecting the operations of the user U of the information terminal 1. The operation unit 12 has, for example, a touch panel, a trackball, or a mouse. When the operation unit 12 is a touch panel, the operation unit 12 is provided so as to overlap the display unit 15. The operation unit 12 notifies the operation reception unit 173 of operation data indicating at least either the position where the operation was performed or the operation amount.

[0045] The position detection unit 13 has a device for detecting the approximate position of the information terminal 1. The position detection unit 13 has, for example, a GPS (Global Positioning System) receiver, and notifies the approximate position specifying unit 174 of position data indicating the latitude and longitude detected based on the radio waves received from GPS satellites. The position detection unit 13 may have at least any one of an altimeter, a compass, or a gyro sensor, and notify the approximate position specifying unit 174 of the altitude of the information terminal 1 and the azimuth in which the information terminal 1 is facing as position data.

[0046] The imaging unit 14 is a device that captures a subject around the information terminal 1 and generates imaging image data, and is, for example, a camera. The imaging unit 14 generates imaging image data by imaging at least a partial area of the space to which the information terminal 1 belongs. The imaging unit 14 may continuously generate imaging image data at a predetermined time interval (frame rate). Further, the imaging unit 14 may have a plurality of imaging elements and be a stereo camera that generates distance data indicating the distance to the subject. In this case, the feature point group data may include information indicating the distance from the information terminal 1 to the subject corresponding to each feature point.

[0047] The display unit 15 is a device for displaying various types of information, and is, for example, a display. The display unit 15 displays, for example, a screen for the user U of the information terminal 1 to perform operations, or an image based on the imaging image data generated by the imaging unit 14. Further, the display unit 15 displays an AR image provided from the information processing device 2 overlaid on an image of the space (for example, an imaging image) visually recognized by the user U.

[0048] The storage unit 16 is a terminal storage unit having a storage medium such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 16 stores a program executed by the control unit 17. Further, the storage unit 16 temporarily stores image data to be displayed on the display unit 15, such as captured image data and AR image data. The storage unit 16 stores a plurality of feature point group data indicating a feature point group of a subject included in a plurality of images created by imaging a space from a plurality of positions and orientations in the space, in association with the positions and orientations in the space.

[0049] The control unit 17 has, for example, a CPU (Central Processing Unit). The control unit 17 functions as a transmission processing unit 171, a reception processing unit 172, an operation reception unit 173, a rough position specifying unit 174, an image processing unit 175, a detailed position specifying unit 176, a display processing unit 177, and a space data acquisition unit 178 by executing a program stored in the storage unit 16.

[0050] The transmission processing unit 171 transmits various data to the information processing device 2 or other external devices via the communication unit 11. The transmission processing unit 171 transmits, for example, operation data received from the operation reception unit 173 to the information processing device 2. The operation data includes operation content data indicating the content of the operation, display coordinate data indicating the position of the operation (that is, the position where the AR image is to be displayed), and display orientation data indicating the orientation for displaying the AR image. The operation data may further include speed data indicating the operation speed of the user U corresponding to the speed at which the AR image is moved.

[0051] When the user U performs an operation to display an AR image on the display unit 15, the transmission processing unit 171 transmits, together with the operation content data, display coordinate data indicating the coordinates in the space when the AR image is displayed on the display unit 15, display orientation data, and speed data to the information processing device 2. Further, the transmission processing unit 171 transmits the rough position data received from the rough position specifying unit 174 to the information processing device 2.

[0052] The transmission processing unit 171 may transmit operation data using Pub / Sub messages for accessing the in-memory database in order to transmit and receive data in real time asynchronously with the in-memory database of the information processing apparatus 2.

[0053] Also, the transmission processing unit 171 acquires feature point group data composed of a plurality of feature points of a subject included in the captured image data from the image processing unit 175, and transmits the acquired feature point group data to the information processing apparatus 2. The transmission processing unit 171 may transmit the captured image data received from the image processing unit 175 to the information processing apparatus 2 instead of or together with the feature point group data. For example, the transmission processing unit 171 writes the feature point group data by transmitting a Pub message to the in-memory database corresponding to the space to which the approximate position specified by the approximate position specifying unit 174 belongs among the plurality of in-memory databases stored in the information processing apparatus 2.

[0054] Also, when there is no registered feature point group data in the plurality of registered feature point group data stored in the storage unit 16 whose difference from the feature point group data is equal to or less than a threshold value, the transmission processing unit 171 transmits the feature point group data acquired from the image processing unit 175 to the information processing apparatus 2 together with an update request for updating the registered feature point group data.

[0055] The reception processing unit 172 receives various data from the information processing apparatus 2 or other external devices via the communication unit 11. For example, the reception processing unit 172 receives display coordinate data transmitted to the information processing apparatus 2 by another information terminal 1 belonging to the same space.

[0056] When the information terminal 1a-1 transmits display coordinate data to the information processing apparatus 2 in response to an operation by the user U1 who uses the information terminal 1a-1 to display an AR image, the reception processing unit 172 of the information terminal 1a-2 receives the display coordinate data transmitted by the coordinate transmission unit 232 of the information processing apparatus 2, which will be described later. The reception processing unit 172 inputs the received display coordinate data to the detailed position specifying unit 176. Further, the reception processing unit 172 receives space data corresponding to the space to which the information terminal 1 belongs from the information processing apparatus 2. The reception processing unit 172 inputs the received space data to the space data acquisition unit 178.

[0057] The operation reception unit 173 specifies the content of the operation performed by the user U based on the operation data notified from the operation unit 12. The operation reception unit 173 creates operation content data indicating the type of the AR image to be displayed based on the received operation. The operation reception unit 173 notifies the information processing apparatus 2 of the content of the operation by inputting the created operation content data to the transmission processing unit 171. For example, when the operation reception unit 173 receives an operation for superimposing an AR image on an image of a space (for example, a captured image) and displaying it on the display unit 15, the operation reception unit 173 creates display coordinate data indicating the coordinates in the space when the AR image is displayed on the display unit 15.

[0058] The operation reception unit 173 may acquire detailed position data indicating the detailed position specified by the detailed position specifying unit 176, and create display coordinate data based on the position and orientation of the information terminal 1 indicated by the acquired detailed position data and the position where the user U operated on the operation unit 12. That is, the operation reception unit 173 may create display coordinate data based on the relationship between the position and orientation specified by the detailed position specifying unit 176 and the position of the operation received by the operation reception unit 173.

[0059] In the case of the example shown in Fig. 3(b), the operation reception unit 173 creates display coordinate data including the X coordinate and the Z coordinate of the position where the hand H touches the operation unit 12. When the position of the hand H moves, the operation reception unit 173 creates display coordinate data at predetermined time intervals. The operation reception unit 173 may create display coordinate data including speed data indicating the speed at which the hand H moves based on the display coordinate data created at a plurality of times. When the imaging unit 14 is a stereo camera capable of generating distance data and the Y coordinate of the subject at the position where the hand H touches in Fig. 3(b) is known, the operation reception unit 173 may create display coordinate data including the Y coordinate. The operation reception unit 173 inputs the created display coordinate data to the transmission processing unit 171.

[0060] Based on the relationship between the position and orientation of the information terminal 1 identified by the detailed position identification unit 176 and the position of the operation received by the operation reception unit 173, the operation reception unit 173 may create display orientation data indicating the orientation of the AR image when the AR image is displayed on the display unit 15. The operation reception unit 173 inputs the display orientation data to the transmission processing unit 171.

[0061] The approximate position identification unit 174 generates approximate position data indicating the approximate position of the information terminal 1 based on the received radio waves. The approximate position identification unit 174 identifies the approximate position of the information terminal 1 based on the position data input from the position detection unit 13. The approximate position data is data indicating the position with an accuracy corresponding to the accuracy of GPS, for example, position data with an error of 10 m or more.

[0062] The approximate position identification unit 174 notifies the information processing apparatus 2 of the approximate position of the information terminal 1 by inputting the identified position data to the transmission processing unit 171. The approximate position identification unit 174 may notify the information processing apparatus 2 of the approximate position data periodically, or may notify the information processing apparatus 2 of the approximate position data when the user U performs a predetermined operation or when it is necessary to acquire the registered feature point group data.

[0063] The image processing unit 175 creates feature point group data composed of a plurality of feature points of the subject included in the captured image data. Specifically, the image processing unit 175 first identifies the contour line of the subject and the contour line of the uneven portion of the subject by identifying pixels in the captured image data whose pixel value change amount with adjacent pixels is equal to or greater than a threshold value. The image processing unit 175 creates feature point group data composed of a plurality of feature points on the identified contour line.

[0064] As shown in the example of FIG. 4, the image processing unit 175 may create feature point group data by identifying points where the direction of the contour line changes as feature points. The image processing unit 175 creates feature point group data every time new captured image data is input from the imaging unit 14. The image processing unit 175 inputs the created feature point group data to the detailed position identification unit 176. When it is necessary for the image processing unit 175 to acquire spatial data from the information processing device 2 (that is, when it is necessary to identify the space to which the information terminal 1 belongs), the image processing unit 175 inputs the feature point group data to the transmission processing unit 171.

[0065] Also, when the image processing unit 175 receives a notification that the detailed position identification unit 176 cannot identify the position and orientation of the information terminal 1 based on the registered feature point group data acquired from the information processing device 2, the image processing unit 175 inputs the created feature point group data to the transmission processing unit 171 to register the feature point group data in the storage device 4, and instructs the transmission processing unit 171 to transmit an update request to the information processing device 2 to update the registered feature point group data.

[0066] The detailed position identification unit 176 identifies the position and orientation corresponding to the registered feature point group data that is most similar to the feature point group data input from the image processing unit 175 among the plurality of registered feature point group data received from the information processing device 2. The detailed position identification unit 176 identifies the position and orientation corresponding to the registered feature point group data that is most similar to the feature point group data indicating the feature point group of the subject included in the captured image data among the plurality of registered feature point group data stored in the storage unit 16.

[0067] Specifically, the detailed position identification unit 176 first compares the feature point group data input from the image processing unit 175 with a plurality of registered feature point group data included in the spatial data acquired from the information processing apparatus 2 and stored in the storage unit 16. The detailed position identification unit 176 calculates a correlation value between the feature point group data input from the image processing unit 175 and each of the plurality of registered feature point group data included in the spatial data, and identifies the registered feature point group data for which the calculated correlation value is maximized.

[0068] Based on the coordinate data (for example, data indicating latitude, longitude, altitude, and azimuth) associated with the identified registered feature point group data, the detailed position identification unit 176 identifies the position and orientation of the information terminal 1. In this way, the detailed position identification unit 176 can identify the position and orientation of the information terminal 1 with higher accuracy than, for example, the approximate position identification unit 174. The detailed position identification unit 176 identifies the position and orientation of the information terminal 1 with an accuracy of, for example, 1 mm. The detailed position identification unit 176 notifies the reception processing unit 172 and the display processing unit 177 of the detailed position data indicating the identified position and orientation.

[0069] The display processing unit 177 causes the display unit 15 to display various types of information or images. For example, the display processing unit 177 causes the display unit 15 to display a captured image based on the captured image data generated by the imaging unit 14 capturing the space to which the information terminal 1 belongs. Further, the display processing unit 177 determines the position and orientation at which to display the AR image based on the display coordinate data. The display processing unit 177 superimposes the AR image on an image of the space that the user U is viewing (for example, a captured image based on the captured image data generated by the imaging unit 14 capturing the space to which the information terminal 1 belongs) at the position in the display unit 15 corresponding to the coordinates in the space indicated by the display coordinate data.

[0070] When the display processing unit 177 performs an operation to display an AR image by the user U, based on the detailed position data input from the detailed position specifying unit 176 and the operation position data notified from the operation reception unit 173, the display processing unit 177 creates display coordinate data indicating the position and orientation for displaying the AR image. When a user U of another information terminal 1 belonging to the same space as the information terminal 1 performs an operation to display an AR image, the display processing unit 177 acquires from the information processing apparatus 2 the display coordinate data used for displaying the AR image corresponding to the operation.

[0071] As shown in FIG. 2, when the information terminals 1a-1 and 1a-2 exist in the first space, the display processing unit 177 of the information terminal 1a-1 displays the AR image at the position and orientation based on the display coordinate data notified from the operation reception unit 173. On the other hand, the display processing unit 177 of the information terminal 1a-2 displays the AR image at the position and orientation based on the display coordinate data notified from the information processing apparatus 2. By operating the display processing units 177 of the information terminals 1a-1 and 1a-2 in this way, the user U1 using the information terminal 1a-1 and the user U2 using the information terminal 1a-2 can simultaneously view AR images with different orientations at different positions according to the position and orientation of the information terminal 1.

[0072] The space data acquisition unit 178 acquires from the information processing apparatus 2 space data including registered feature point group data via the communication unit 11 and the reception processing unit 172. In response to the operation reception unit 173 receiving an operation (for example, an operation to start a predetermined application software, an operation to log in to a predetermined site) for acquiring a plurality of registered feature point group data from the information processing apparatus, the space data acquisition unit 178 acquires from the information processing apparatus space data including a plurality of registered feature point group data corresponding to the space to which the information terminal 1 belongs, and stores the acquired space data in 16.

[0073] The spatial data acquisition unit 178 notifies the detailed position identification unit 176 of the acquired plurality of registered feature point group data. For example, the spatial data acquisition unit 178 stores the acquired plurality of registered feature point group data in the storage unit 16, and notifies the detailed position identification unit 176 of the plurality of registered feature point group data via the storage unit 16.

[0074] (Configuration of the information processing apparatus 2) FIG. 8 is a diagram showing the configuration of the information processing apparatus 2. The information processing apparatus 2 includes a communication unit 21, a storage unit 22, and a control unit 23. The control unit 23 includes a coordinate reception unit 231, a coordinate transmission unit 232, a terminal data reception unit 233, a feature point group transmission unit 234, an information reception unit 235, a spatial data transmission unit 236, a spatial data management unit 237, and a unit number management unit 238.

[0075] The communication unit 21 has a communication interface for connecting to a communication network. The communication unit 21 transmits and receives various data to and from the information terminal 1, the computer group 3, and the storage device 4.

[0076] The storage unit 22 is a device storage unit having a ROM, a RAM, and an SSD (Solid State Drive). The storage unit 22 stores programs executed by the control unit 23. The storage unit 22 also stores a first space database 221 and a second space database 222 as a plurality of in-memory databases corresponding to a plurality of spaces. The storage unit 22 stores, for example, a plurality of in-memory databases (for example, the first space database 221 and the second space database 222) that store display coordinate data or feature point group data corresponding to a plurality of spaces (for example, the first space and the second space).

[0077] As described above, an in-memory database is a database constructed on the main memory and is a database that can be accessed at a higher speed compared to a database stored in a disk storage mechanism. In FIG. 8, the storage unit 22 exemplifies a database for the first space 221 and a database for the second space 222, but the storage unit 22 stores databases corresponding to more spaces.

[0078] The storage unit 22 stores the first information terminal and the second information terminal belonging to the space of the same coordinate system as a plurality of information terminals 1 that display AR images within the same space. The first information terminal is the information terminal 1 for which the user U has performed an operation to display an AR image, and is the information terminal 1a-1 in the example shown in FIG. 2. The second information terminal is another information terminal 1 belonging to the same space as the first information terminal, and is the information terminal 1a-2 in the example shown in FIG. 2. The storage unit 22 stores a plurality of terminal identification information for identifying the plurality of information terminals 1 included in the space in association with each of the plurality of spaces.

[0079] Specifically, in the in-memory database stored in the storage unit 22, space identification information for identifying the space to which each of the plurality of information terminals 1 belongs is registered in association with the terminal identification information for identifying each of the plurality of information terminals 1. Further, in the in-memory database, in association with the space identification information, AR image identification information for identifying the AR image to be displayed on the information terminal 1 belonging to the space corresponding to the space identification information and display coordinate data indicating the position and orientation for displaying the AR image are registered.

[0080] Multiple in-memory databases may store the feature point group data received from the information terminal 1 belonging to the corresponding space in order to transfer the feature point group data to the computer group 3. When the first space database 221 corresponding to the first space receives the feature point group data from one or more information terminals 1 belonging to the first space, it transfers the received feature point group data to the computer 30 that analyzes whether the feature point group data matches the registered feature point group data corresponding to the first space. The in-memory database may relay the feature point group data between the information terminal 1 and the computer group 3 based on the Pub / Sub message sent from the information terminal 1 or the computer group 3 without going through the control unit 23.

[0081] The control unit 23 has, for example, a CPU. By executing the program stored in the storage unit 22, the control unit 23 functions as a coordinate reception unit 231, a coordinate transmission unit 232, a terminal data reception unit 233, a feature point group transmission unit 234, an information reception unit 235, a space data transmission unit 236, a space data management unit 237, and a unit number management unit 238.

[0082] The coordinate reception unit 231 receives the operation content data and the display coordinate data transmitted by the information terminal 1 in response to the operation of the user U using the information terminal 1, and registers the received operation content data and display coordinate data in the in-memory database. The coordinate reception unit 231 registers the display coordinate data in the in-memory database corresponding to the space in which the information terminal 1 that transmitted the display coordinate data is included. For example, when the coordinate reception unit 231 receives the display coordinate data from the information terminal 1a-1 belonging to the first space, it stores the display coordinate data in the first space database 221.

[0083] In response to the display coordinate data being registered in the in-memory database by the coordinate reception unit 231, the coordinate transmission unit 232 transmits the display coordinate data to a second information terminal different from the first information terminal among the plurality of information terminals 1. The coordinate transmission unit 232 transmits the display coordinate data registered in the in-memory database corresponding to the space in which the second information terminal is included to the second information terminal.

[0084] By referring to the storage unit 22, the coordinate transmission unit 232 transmits the display coordinate data to a second information terminal having second terminal identification information associated with a space corresponding to the first terminal identification information of the first information terminal that transmitted the display coordinate data. The coordinate transmission unit 232 may transmit the display orientation data to the second information terminal in response to the display orientation data transmitted by the first information terminal being registered in the in-memory database. As a result, a plurality of information terminals 1 belonging to the same space share the position and display orientation for displaying the AR image.

[0085] Further, in response to the operation content data received from the first information terminal being registered in the in-memory database, the coordinate transmission unit 232 transmits data corresponding to a second AR image corresponding to the operation content data or data for identifying the second AR image to the second information terminal. The second AR image is an image corresponding to the first AR image, showing the same object and having a different orientation or size. As a result, a plurality of information terminals 1 belonging to the same space share the same AR image.

[0086] The coordinate transmission unit 232 transmits the display coordinate data and the AR image stored in the first space database 221 to the information terminal 1a-2. When the coordinate transmission unit 232 has already transmitted the AR image to the information processing apparatus 2, the coordinate transmission unit 232 may transmit, instead of the AR image, AR image identification information for identifying the AR image to be displayed on the information terminal 1a-2 among the plurality of AR images already transmitted.

[0087] The terminal data reception unit 233 receives the feature point group data from the information terminal 1 via the communication unit 21. Further, the terminal data reception unit 233 receives the approximate position data indicating the approximate position of the information terminal 1 from the information terminal 1 that transmitted the feature point group data. The terminal data reception unit 233 notifies the received feature point group data and approximate position data to the feature point group transmission unit 234. The terminal data reception unit 233 may store the feature point group data and the approximate position data in the storage unit 22 and notify the feature point group data and the approximate position data to the feature point group transmission unit 234 via the storage unit 22.

[0088] The feature point group transmission unit 234 transmits the feature point group data received from the information terminal 1 to the computer group 3. For example, as shown in FIG. 5, when the terminal data reception unit 233 receives the feature point group data transmitted by the information terminal 1 together with the activation information, the feature point group transmission unit 234 transmits the feature point group data to the computer group 3 together with a message requesting spatial data.

[0089] When the feature point group transmission unit 234 acquires the approximate position data from the terminal data reception unit 233, the feature point group transmission unit 234 may select one or more computers 30 corresponding to a space including the approximate position indicated by the approximate position data from among the plurality of computers 30 included in the computer group 3, and transmit the feature point group data to the selected one or more computers 30. That is, the feature point group transmission unit 234 may transmit the feature point group data to the computer 30 associated with the space corresponding to the data to which the approximate position indicated by the approximate position data transmitted by the information terminal 1 together with the feature point group data belongs.

[0090] The feature point group transmission unit 234 may relay the feature point group data between the information terminal 1 and the computer group 3 via in-memory databases corresponding to the respective plurality of spaces. The feature point group transmission unit 234 transmits the feature point group data to the computer 30 associated with the space corresponding to the in-memory database in which the feature point group data is written among the plurality of in-memory databases.

[0091] The information receiving unit 235 receives, from the computer 30 that has identified the space to which the information terminal 1 belongs among one or more computers 30 to which the feature point group transmitting unit 234 has transmitted the feature point group data, predetermined information indicating that the space to which the information terminal 1 belongs has been identified. Specifically, the information receiving unit 235 receives, from one of the one or more computers 30 that have transmitted the feature point group data, predetermined information indicating that the correlation value between any one of the plurality of registered feature point group data stored in the computer 30 and the feature point group data transmitted by the feature point group transmitting unit 234 is equal to or greater than a threshold value. That is, the information receiving unit 235 receives, from the computer 30, predetermined information indicating that the feature point group data transmitted by the feature point group transmitting unit 234 is equivalent to the registered feature point group data of the subject included in the space for which the computer 30 is responsible for the matching process.

[0092] The information receiving unit 235 may acquire space data including a plurality of registered feature point group data corresponding to the space for which the computer 30 is responsible for the matching process, together with the predetermined information. The information receiving unit 235 inputs the acquired space data to the space data transmitting unit 236, together with the terminal identification information of the information terminal 1 that created the feature point group data transmitted by the feature point group transmitting unit 234 to the computer group 3. Further, the information receiving unit 235 acquires space identification information for identifying the space for which the computer 30 is responsible for the matching process, and may store the space identification information in the storage unit 22 in association with the terminal identification information of the information terminal 1 that created the feature point group data transmitted by the feature point group transmitting unit 234 to the computer group 3.

[0093] The space data transmitting unit 236 transmits the plurality of registered feature point group data included in the space data corresponding to the computer 30 that has transmitted the predetermined information to the information terminal 1. The space data transmitting unit 236 transmits the plurality of registered feature point group data to the information terminal 1 corresponding to the terminal identification information acquired from the information receiving unit 235.

[0094] The spatial data management unit 237 performs management to maintain the spatial data stored in the computer group 3 or the storage device 4 in the latest state. The spatial data management unit 237 maintains the spatial data in the latest state, for example, by associating the feature point group data received from the information terminal 1 with positions in the space and storing it in the storage device 4.

[0095] When the spatial data management unit 237 determines that the feature point group data received from the information terminal 1 is the latest feature point group data, it stores the feature point group data received from the information terminal 1 in the storage device 4 as registered feature point group data. For example, when the spatial data management unit 237 receives the feature point group data together with an update request for requesting an update of the registered feature point group data from a plurality of information terminals 1, it refers to, for example, the timestamps included in the feature point group data, and determines that the feature point group data based on the captured image data generated at a later time point is the latest feature point group data.

[0096] When the device reception processing unit receives a plurality of feature point group data from a plurality of information terminals 1 that have transmitted detailed position data indicating the same position and orientation, the spatial data management unit 237 may store the feature point group data with a larger data amount in the storage device 4 as the registered feature point group data. The data amount is, for example, the number of point groups included in the feature point group data. It is considered that the finer features of the subject belonging to the space are represented in the feature point group data as the number of point groups is larger. Therefore, when the spatial data management unit 237 registers such feature point group data, the accuracy of specifying the position and orientation by the information terminal 1 using the registered feature point group data is improved.

[0097] When the feature point group data associated with the position and orientation indicated by the detailed position data transmitted by the information terminal 1 is not stored in the storage device 4 together with the feature point group data, the spatial data management unit 237 may store the feature point group data received from the information terminal 1 in the storage device 4 as registered feature point group data. The storage unit 22 stores the latitude, longitude, altitude, and azimuth corresponding to each of the plurality of registered feature point group data stored in the storage device 4. When the latitude, longitude, altitude, and azimuth indicated by the detailed position data are not stored in the storage unit 22, the spatial data management unit 237 stores the feature point group data in the storage device 4.

[0098] When the registered feature point group data corresponding to the position and orientation indicated by the position data is stored in the storage device 4, the spatial data management unit 237 may store the feature point group data received from the information terminal 1 in the storage device 4 as registered feature point group data on the condition that the data amount of the feature point group data received from the information terminal 1 is larger than the data amount of the stored registered feature point group data.

[0099] The storage unit 22 stores the data amount of each of the plurality of registered feature point group data stored in the storage device 4 together with the latitude, longitude, altitude, and azimuth. The spatial data management unit 237 compares the data amount of the feature point group data transmitted from the information terminal 1 with the data amount of the registered feature point group data stored in the storage unit 22 in association with the position and orientation indicated by the detailed position data transmitted from the information terminal 1. When the comparison result shows that the data amount of the feature point group data received from the information terminal 1 is larger, the spatial data management unit 237 stores the feature point group data received from the information terminal 1 in the storage device 4. By operating in this way, the spatial data management unit 237 can prevent the registered feature point group data from deteriorating by updating the registered feature point group data.

[0100] The spatial data management unit 237 may store the feature point group data in the storage device 4 on the condition that the feature point group data received from the information terminal 1 includes data indicating a subject having a shape registered in advance. The subject having a shape registered in advance is, for example, a newly constructed building. By operating in this way, when it is known in advance that there is no registered feature point group data, the registered feature point group data can be registered promptly.

[0101] The number management unit 238 determines the number of a plurality of computers 30 that execute a matching process for specifying the space to which the information terminal 1 belongs based on a preset condition. The number management unit 238 changes the number of the plurality of computers 30, for example, based on the amount of the registered feature point group data stored in the storage device 4. The feature point group transmission unit 234 transmits the feature point group data received from the plurality of information terminals 1 to the computers 30 in the number determined by the number management unit 238. By operating in this way, even when the amount of the registered feature point group data increases over time, an increase in the time required for the matching process can be suppressed.

[0102] The number management unit 238 may change the number of the plurality of computers 30 based on the time required from when the feature point group transmission unit 234 transmits the feature point group data to the plurality of computers 30 until a predetermined piece of information indicating that the space to which the information terminal 1 belongs has been specified is acquired (that is, the time required for the matching process). When the time required from when the feature point group transmission unit 234 transmits the feature point group data to the plurality of computers 30 until the predetermined piece of information is acquired is long, it is considered that the number of the computers 30 is insufficient. Therefore, when the time becomes equal to or greater than a threshold value, the number management unit 238 increases the number of the computers 30 that execute the matching process, thereby suppressing an excessive increase in the time required for the matching process.

[0103] Incidentally, when the number of information terminals 1 from which the information processing apparatus 2 receives feature point group data increases, the amount of feature point group data received by the terminal data reception unit 233 increases. When the amount of feature point group data increases, the time required for the matching process becomes longer. Therefore, the number management unit 238 may change the number of a plurality of computers 30 based on the amount of feature point group data received by the terminal data reception unit 233. The number management unit 238 may change the number of a plurality of computers based on the number of information terminals 1 that transmit feature point group data.

[0104] When the amount of feature point group data received by the terminal data reception unit 233 within a predetermined time exceeds a threshold value, or when the number of information terminals 1 that receive feature point group data within a predetermined time increases, the number management unit 238 increases the number of computers 30 that execute the matching process, thereby suppressing the situation where the time required for the matching process becomes too long. The number management unit 238 may increase the number of computers 30 that execute the matching process when the number of information terminals 1 from which the terminal data reception unit 233 receives feature point group data exceeds a threshold value.

[0105] When the information terminal 1 is moving, it is assumed that the information terminal 1 moves to a different space and the frequency of the matching process for identifying the space to which the information terminal 1 belongs increases. Therefore, the number management unit 238 may change the number of a plurality of computers 30 based on whether the approximate position data received by the terminal data reception unit 233 from the information terminal 1 indicates that the information terminal 1 is moving.

[0106] When the number management unit 238 identifies that the number of moving information terminals 1 is equal to or more than a predetermined number, it increases the number of computers 30 that execute the matching process, thereby suppressing the situation where the time required for the matching process becomes too long. Conversely, when the number management unit 238 identifies that the number of moving information terminals 1 is less than a predetermined number, it can reduce the usage cost of the computers 30 by decreasing the number of computers 30 that execute the matching process.

[0107] The number management unit 238 identifies the moving speed of the information terminal 1 based on a plurality of approximate position data transmitted at a plurality of times, and increases the number of computers 30 when the number of information terminals 1 with a moving speed equal to or higher than a threshold value is equal to or more than a predetermined number. By operating in this way, the number management unit 238 can change the number of computers 30 based on the number of information terminals 1 with a high probability of space switching, so that the number of computers 30 for executing the matching process can be made more appropriate.

[0108] (Configuration of computer group 3) FIG. 9 is a diagram showing the configuration of a computer 30 included in the computer group 3. The communication unit 31 includes a communication unit 31, a storage unit 32, and a control unit 33. The control unit 33 includes a feature point group acquisition unit 331, a registration data acquisition unit 332, and a data analysis unit 333.

[0109] The communication unit 31 has a communication interface for connecting to a communication network. The communication unit 31 transmits and receives various data to and from the information processing device 2 and the storage device 4.

[0110] The storage unit 32 includes a ROM, a RAM, and an SSD. The storage unit 32 stores programs executed by the control unit 33.

[0111] The control unit 33 includes, for example, a CPU. The control unit 33 functions as a feature point group acquisition unit 331, a registration data acquisition unit 332, and a data analysis unit 333 by executing programs stored in the storage unit 32.

[0112] The feature point group acquisition unit 331 acquires feature point group data from the information processing device 2. The feature point group acquisition unit 331 inputs the acquired feature point group data to the data analysis unit 333.

[0113] The registration data acquisition unit 332 acquires a plurality of registered feature point group data by reading out the plurality of registered feature point group data corresponding to the space assigned to the computer 30 from the storage device 4 that stores the registered feature point group data corresponding to a plurality of spaces. The registration data acquisition unit 332 inputs the acquired plurality of registered feature point group data to the data analysis unit 333. The registration data acquisition unit 332 may store the acquired plurality of registered feature point group data in the storage unit 32.

[0114] Note that the space identification information corresponding to the space assigned to the computer 30 is stored in, for example, the storage unit 32, and the registration data acquisition unit 332 acquires, from the storage device 4, the plurality of registered feature point group data stored in the storage device 4 in association with the space identification information.

[0115] When the difference between the feature point group data acquired by the feature point group acquisition unit 331 and any one of the plurality of registered feature point group data acquired by the registration data acquisition unit 332 corresponding to the space assigned to each computer 30 is equal to or less than a threshold value, the data analysis unit 333 transmits predetermined information to the information processing device 2. As described above, the predetermined information is information indicating that the feature point group data received by the registration data acquisition unit 332 is composed of the feature point group of the subject included in the space for which the computer 30 is responsible for the matching process.

[0116] [Flow of processing in the information processing system S] [Processing for acquiring spatial data] FIG. 10 is a sequence diagram showing the flow until a plurality of information terminals 1 belonging to the same space acquire spatial data. The sequence diagram shown in FIG. 10 starts from the point in time when the information terminal 1 needs to acquire spatial data. The point in time when the need to acquire spatial data arises is, for example, when the user U starts a predetermined application software on the information terminal 1, logs in to a predetermined site, moves a predetermined distance or more from the point where the spatial data was last acquired, or when a predetermined time or more has elapsed since the time when the spatial data was last acquired.

[0117] The first information terminal and the second information terminal in FIG. 10 belong to the same first space, for example, the information terminals 1a-1 and 1a-2 shown in FIG. 1. The first information terminal transmits first approximate position data indicating the position of the first information terminal and first feature point group data generated based on the captured image data to the information processing device 2. The information processing device 2 transmits the first feature point group data to a first computer that executes a matching process using the registered feature point group data of the first space including the approximate position indicated by the received first approximate position data. Here, the information processing device 2 may transmit the first feature point group data to a plurality of first computers.

[0118] When the first computer determines that the first feature point group data matches any of the plurality of registered feature point group data, the first computer determines that the first information terminal is in the first space corresponding to the matched registered feature data, and transmits first space data including the plurality of registered feature point group data corresponding to the first space to the information processing device 2. The information processing device 2 transmits the received first space data to the first information terminal. Thereafter, the first information terminal identifies its detailed position by comparing the feature point group data generated by itself with the plurality of received registered feature point group data.

[0119] The second information terminal operates in the same manner as the first information terminal, and transmits second approximate position data indicating the position of the second information terminal and second feature point group data generated based on the captured image data to the information processing device 2. Since the second information terminal also belongs to the same first space as the first information terminal, the information processing device 2 transmits the second feature point group data to the first computer and receives the first space data from the first computer. The information processing device 2 transmits the first space data to the second information terminal, and the second information terminal identifies its detailed position by comparing the feature point group data generated by itself with the plurality of registered feature point group data included in the received first space data.

[0120] FIG. 11 is a sequence diagram showing the flow until a plurality of information terminals 1 belonging to different spaces acquire spatial data. The first information terminal and the third information terminal in FIG. 11 belong to different spaces, for example, the information terminals 1a-1 and 1b-1 shown in FIG. 1. The operation of the first information terminal is the same as the operation shown in FIG. 10.

[0121] The third information terminal transmits third schematic position data indicating the position of the third information terminal and third feature point group data generated based on the captured image data to the information processing device 2. Since the third information terminal belongs to a second space different from the first information terminal, the information processing device 2 transmits the third feature point group data to the second computer in charge of the second space corresponding to the third schematic position data, and receives the second space data from the third computer. The information processing device 2 transmits the second space data to the second information terminal, and the second information terminal specifies its detailed position by comparing the feature point group data generated by itself with a plurality of registered feature point group data included in the received second space data.

[0122] (Process of displaying an AR image) FIG. 12 is a sequence diagram showing the flow of the process in which a plurality of information terminals 1 belonging to the same space display an AR image. The sequence diagram shown in FIG. 12 starts from the time when the user U1 using the first information terminal performs an operation to display an AR image. The first information terminal specifies the content of the operation performed by the user U1 (S11), and specifies the operation position coordinates indicating the position where the operation is performed (S12). The first information terminal transmits operation content data indicating the content of the operation and coordinate data indicating the operation position to the information processing device 2.

[0123] Based on the received operation content data, the information processing device 2 selects AR image data to be displayed on the first information terminal, and transmits the selected AR image data to the first information terminal. Note that when the first information terminal stores the AR image data, the information processing device 2 does not have to transmit the AR image data to the first information terminal.

[0124] The information processing device 2 transmits display coordinate data corresponding to the coordinate data received from the first information terminal to the second information terminal belonging to the same space as the first information terminal. Further, the information processing device 2 also transmits the AR image data transmitted to the first information terminal to the second information terminal. The first information terminal and the second information terminal display an AR image based on the received AR image data at a position and orientation corresponding to the display coordinate data (S13, S14). By operating in this way, the first information terminals 1 belonging to the same space can display the same AR image at positions and orientations corresponding to their respective positions and orientations.

[0125] (Registration of Spatial Data) FIG. 13 is a sequence diagram showing the flow of processing in which the information terminal 1 registers the feature point group data as spatial data. The sequence diagram shown in FIG. 13 starts from the point in time when the first information terminal compares the feature point group data created by itself with the registered feature point group data.

[0126] If the first information terminal determines that there is no spatial data including registered feature point group data that matches the first feature point group data created by itself (YES in S21), it determines that it is necessary to update the spatial data (S22), and transmits the first feature point group data to the information processing device 2 together with an update request. Similarly, if the second information terminal determines that there is no spatial data including registered feature point group data that matches the second feature point group data created by itself (YES in S23), it transmits the second feature point group data to the information processing device 2 together with an update request for updating the spatial data.

[0127] When the information processing device 2 receives the first feature point group data and the second feature point group data corresponding to the same position and orientation, it determines which feature point group data should be registered based on any of the above-described predetermined conditions, and selects one of the feature point group data (S24). The information processing device 2 transmits the selected feature point group data to the storage device 4, and the storage device 4 registers the received feature point group data as spatial data (S25).

[0128] [Modification Example] In the above description, the information processing apparatus 2 relays the feature point group data and the registered feature point group data between the information terminal 1 and the computer group 3. However, the computer group 3 may have the functions of the information processing apparatus 2.

[0129] Also, in the above description, the case where the information processing apparatus 2 provides the AR image to the information terminal 1 in response to the information terminal 1 transmitting the display coordinate data to the information processing apparatus 2 is exemplified. However, the information terminal 1 may store a plurality of AR images in advance. In this case, the information terminal 1 selects an AR image to be displayed from the plurality of stored AR images based on an instruction from the information processing apparatus 2, and displays the selected AR image at the position and orientation based on the received display coordinate data.

[0130] Also, in the above description, the case where the information processing apparatus 2 has an in-memory database and exchanges various display coordinate data and feature point group data via the in-memory database is exemplified. However, the information processing apparatus 2 may not have an in-memory database, and a plurality of information terminals 1 may exchange these data via another storage medium.

[0131] [Effect by the information processing system S] The information processing apparatus 2 according to the present embodiment receives the display coordinate data transmitted by the first information terminal, registers the received display coordinate data in the in-memory database, and transmits the display coordinate data to the second information terminal belonging to the same space as the first information terminal in response to the display coordinate data being registered in the in-memory database. By the information processing apparatus 2 transmitting and receiving the display coordinate data to and from a plurality of information terminals 1 via the in-memory database, the time required for the information terminal 1 to display the AR image can be shortened.

[0132] In addition, the computer group 3 acquires feature point group data based on the captured image data generated by the information terminal 1, and a plurality of computers 30 each responsible for a different space compare the plurality of registered feature point group data with the acquired feature point group data to identify the space to which the information terminal 1 belongs. Then, the computer group 3 provides the information terminal 1 with a plurality of registered feature point group data corresponding to the identified space. By operating in this way, the computer group 3 enables the information terminal 1 to limit the plurality of registered feature point group data to be compared with the feature point group data created by itself to the plurality of registered feature point group data corresponding to the space to which it belongs, so that the accurate position and orientation of the information terminal 1 can be identified in a short time.

[0133] In addition, the information processing device 2 changes the number of computers 30 that execute the matching process with the feature point group data generated by the information terminal 1 based on preset conditions. By operating in this way, even if the number of information terminals 1 increases or decreases, or the amount of feature point group data transmitted by the information terminal 1 increases or decreases, the time required to transmit appropriate registered feature point group data to the information terminal 1 can be maintained within a certain range.

[0134] In addition, when there is no registered feature point group data corresponding to the feature point group data generated by the information terminal 1 itself, the information terminal 1 can register the feature point group data generated by itself in the storage device 4. By registering the feature point group data generated by a large number of information terminals 1 in the storage device 4, the registered feature point group data stored in the storage device 4 can be maintained in the latest state. Furthermore, when the information processing device 2 receives update requests for feature point group data corresponding to the same position and orientation from a plurality of information terminals 1, it registers more reliable feature point group data in the storage device 4. By operating in this way, the reliability of the registered feature point group data stored in the storage device 4 is improved.

[0135] As described above, the present invention has been explained using embodiments. However, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist. For example, all or part of the device can be configured by being functionally or physically distributed and integrated in arbitrary units. Also, new embodiments resulting from any combination of a plurality of embodiments are included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination have the effects of the original embodiments combined.

Explanation of Reference Numerals

[0136] 1 Information terminal 2 Information processing device 3 Computer group 4 Storage device 11 Communication unit 12 Operation unit 13 Position detection unit 14 Imaging unit 15 Display unit 16 Storage unit 17 Control unit 21 Communication unit 22 Storage unit 23 Control unit 30 Computer 31 Communication unit 32 Storage unit 33 Control unit 171 Transmission processing unit 172 Reception processing unit 173 Operation reception unit 174 Approximate position specifying unit 175 Image processing unit 176 Detailed position specifying unit 177 Display processing unit 178 Spatial data acquisition unit 221 Database for the first space 222 Database for the second space 231 Coordinate reception unit 232 Coordinate transmission unit 233 Terminal data reception unit 234 Feature point group transmission unit 235 Information reception unit 236 Spatial data transmission unit 237 Spatial data management unit 238 Quantity management unit 331 Feature point group acquisition unit 332 Registered data acquisition unit 333 Data analysis unit

Claims

1. An information terminal, an information processing apparatus capable of communicating with the information terminal, and a storage device, wherein the information terminal has an imaging unit that generates imaging image data by imaging at least a part of a space to which the information terminal belongs, and a terminal transmission processing unit that transmits feature point group data constituted by a plurality of feature point groups of a subject included in the imaging image data to the information processing apparatus, and the information processing apparatus has a terminal data receiving unit that receives the feature point group data and position data indicating the position and orientation of the information terminal from the information terminal, and a space data management unit that stores the feature point group data received from the information terminal in the storage device in association with a position in the space, wherein when the space data management unit determines that the feature point group data associated with the position and orientation indicated by the position data is not stored in the storage device and the feature point group data received from the information terminal is the latest feature point group data, the space data management unit stores the feature point group data received from the information terminal in the storage device as registered feature point group data. An information processing system.

2. When the terminal data receiving unit receives a plurality of pieces of the feature point group data from a plurality of the information terminals that have transmitted the position data indicating the same position and orientation, the space data management unit stores the feature point group data having a larger data amount in the storage device as the registered feature point group data. The information processing system according to claim 1.

3. When the registered feature point group data corresponding to the position and orientation indicated by the position data is stored in the storage device, the space data management unit stores the feature point group data received from the information terminal in the storage device as the registered feature point group data on condition that the data amount of the feature point group data received from the information terminal is larger than the data amount of the stored registered feature point group data. The information processing system according to claim 1 or 2.

4. An information terminal, an information processing apparatus capable of communicating with the information terminal, and a storage device, wherein the information terminal has an imaging unit that generates imaging image data by imaging at least a part of a space to which the information terminal belongs, and a terminal transmission processing unit that transmits feature point group data constituted by a plurality of feature point groups of a subject included in the imaging image data to the information processing apparatus, and the information processing apparatus ​ ​ ​ A terminal data receiving unit that receives the feature point group data from the information terminal; A spatial data management unit that associates the feature point group data received from the information terminal with positions in the space and stores it in the storage device; It has: When the spatial data management unit determines that the feature point group data received from the information terminal is the latest feature point group data, the spatial data management unit stores the feature point group data in the storage device on the condition that the feature point group data received from the information terminal includes data indicating a subject with a pre-registered shape. An information processing system.

5. An information processing system comprising an information terminal, an information processing device capable of communicating with the information terminal, and a storage device. The information terminal: An imaging unit that generates imaging image data by imaging at least a partial area of the space to which the information terminal belongs; A terminal transmission processing unit that transmits feature point group data composed of a plurality of feature point groups of a subject included in the imaging image data to the information processing device; It has: The information processing device: A terminal data receiving unit that receives the feature point group data from the information terminal; A spatial data management unit that associates the feature point group data received from the information terminal with positions in the space and stores it in the storage device; It has: When the spatial data management unit determines that the feature point group data received from the information terminal is the latest feature point group data, the spatial data management unit stores the feature point group data received from the information terminal in the storage device as registered feature point group data. The information terminal further has a terminal storage unit that stores a plurality of the registered feature point group data stored in the storage device. When there is no registered feature point group data in the plurality of registered feature point group data stored in the terminal storage unit, the difference from the feature point group data is less than or equal to a threshold value, the terminal transmission processing unit transmits the feature point group data to the information processing device. An information processing system.

6. A step in which the information terminal generates imaging image data by imaging at least a partial area of the space to which the information terminal belongs; A step of transmitting, from the information terminal to the information processing device, feature point group data composed of a plurality of feature point groups of a subject included in the imaging image data generated by the information terminal and position data indicating the position and orientation of the information terminal. When the information processing apparatus determines that the feature point group data associated with the position and orientation indicated by the position data is not stored in the storage device and that the feature point group data received from the information terminal is the latest feature point group data, a step of associating the feature point group data with a position in the space and storing the same in the storage device; An information processing method comprising the above.

7. A step in which an information terminal generates captured image data by capturing at least a partial area of a space to which the information terminal belongs; A step of transmitting, from the information terminal to an information processing apparatus, feature point group data constituted by a plurality of feature point groups of a subject included in the captured image data generated by the information terminal; When the information processing apparatus determines that the feature point group data received from the information terminal is the latest feature point group data, a step of associating the feature point group data received from the information terminal with a position in the space and storing the same in a storage device on condition that data indicating a subject having a previously registered shape is included in the feature point group data; An information processing method comprising the above.

Images