terminal

Abstract

To display an image of a virtual object while considering a plurality of self-position determination methods.SOLUTION: A determination section 114 that determines a first self-position by a first self-position determination method and determines a second self-position by a second self-position determination method that is less accurate than the first self-position determination method, and a display control section 119 that causes the display 15 to display a first display image DI [1] in which first arranged image AI [1] in which a first virtual object VO [1] is arranged at a position determined based on the first self-position is superimposed on the captured image PI captured by the imaging device 13, and causes the display 15 to display a second display image DI [2] in which second arranged image AI [2] in which a second virtual object VO [2] is arranged at a position determined based on the second self-position is superimposed on the captured image PI.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to a terminal device. [Background technology]

[0002] Patent Document 1 discloses a technology for acquiring location information of a mobile terminal and superimposing an image of a virtual object such as a character on an image obtained by capturing an image of the scenery around the user in accordance with the location information.

[0003] Incidentally, some virtual objects require high accuracy in terms of their position relative to real objects, while others can be placed with low accuracy without causing any problems. The accuracy of the position at which a virtual object is placed depends on the accuracy of the mobile device's own position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-220246 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology, the self-position of the mobile terminal is determined only by the GPS (Global Positioning System), and there is a problem in that a plurality of self-position determination methods is not taken into consideration.

[0006] Therefore, an object of the present invention is to provide a terminal device that displays an image of a virtual object while taking into consideration a plurality of self-positioning methods. [Means for solving the problem]

[0007] A preferred aspect of the present invention provides a terminal device including: a determination unit that determines a first self-location using a first self-location determination method and determines a second self-location using a second self-location determination method that is less accurate than the first self-location determination method; a management unit that manages a first virtual object and a second virtual object that can be positioned with less accuracy than the first virtual object; and a display control unit that causes a display device to display a first display image in which a first placement image, in which the first virtual object is placed at a position determined based on the first self-location, is superimposed on an image captured by an imaging device, and causes the display device to display a second display image in which a second placement image, in which the second virtual object is placed at a position determined based on the second self-location, is superimposed on the image captured by an imaging device. [Effects of the Invention]

[0008] According to the present invention, it is possible to display an image of a virtual object while taking into consideration a plurality of self-positioning methods. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the overall configuration of an information processing system 1. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a location information server 30. [Figure 3] 1 is a table showing an example of the configuration of the landmark database ODB. [Figure 4] FIG. 10 is a diagram showing a method for calculating first position information when using the landmark database ODB. [Figure 5] FIG. 2 is a block diagram showing an example of the configuration of a content server 50. [Figure 6] A table showing an example of the configuration of the object database CDB[1]. [Figure 7] FIG. 2 is a diagram showing an example of a display image DI1. [Figure 8] FIG. 10 is a diagram showing an example of a display image DI2. [Figure 9] FIG. 10 is a diagram showing an example of a display image DI3. [Figure 10] 10 is a table showing an example of the configuration of a user database UDB. [Figure 11] FIG. 1 is a block diagram showing an example configuration of a terminal device 10[k]. [Figure 12] A table showing an example of the configuration of the object database CDB[2]. [Figure 13] FIG. 3 is a functional block diagram of a determination unit 114. [Figure 14] FIG. 2 is a diagram showing an example of an object map CM. [Figure 15] FIG. 4 is a diagram showing an example of an object list CL. [Figure 16] 10 is a flowchart showing an example of the operation of the terminal device 10[k]. DETAILED DESCRIPTION OF THE INVENTION

[0010] 1: First embodiment Hereinafter, the configuration of an information processing system 1 including a terminal device 10 according to a first embodiment of the present invention will be described with reference to FIGS.

[0011] 1-1: Configuration of the first embodiment 1-1-1: Overall structure Fig. 1 shows the overall configuration of an information processing system 1. As shown in Fig. 1, the information processing system 1 includes terminal devices 10[1], 10[2], ... 10[k], ... 10[j], a location information server 30, and a content server 50. j is an integer equal to or greater than 1. k is an integer equal to or greater than 1 and equal to or less than j. In this embodiment, terminal devices 10[1] to 10[j] have the same configuration. However, terminal devices with different configurations may be included.

[0012] In the information processing system 1, terminal devices 10[1] to 10[j], a location information server 30, and a content server 50 are communicably connected to each other via a communication network NET. In FIG. 1, a user U uses the terminal device 10. Furthermore, a user U[k] uses the terminal device 10[k]. In the following, the user U[k] may be described as a representative example of the user U. The terminal device 10[k] may be described as a representative example of the terminal device 10.

[0013] The location information server 30 provides location information to the terminal devices 10[1] to 10[j].

[0014] Specifically, the terminal device 10[k] includes a distance sensor 19, as described below. The terminal device 10[k] transmits data indicating the spatial structure detected by the distance sensor 19 to the location information server 30. The location information server 30 generates VPS (Visual Positioning System) information based on the data indicating the spatial structure received from the terminal device 10[k]. The VPS information indicates the position of the terminal device 10[k] and the direction in which the terminal device 10[k] is facing. The location information server 30 transmits the generated VPS information to the terminal device 10[k] as first location information. The terminal device 10[k], which is the destination of the VPS information generated by the location information server 30 as the first location information, may be the same as or different from the terminal device 10[x] that transmits the data indicating the spatial structure detected by the distance sensor 19 of the terminal device 10[x]. At the time when the VPS information generated by the location information server 30 is transmitted to the terminal device 10[k] as the first location information, i.e., before the user U experiences the content, the terminal device 10[x] is used to prepare for content installation, and data indicating the spatial structure is transmitted to the location information server 30. In this case, the terminal device 10[k] of the user U who experiences the content is different from the terminal device 10[x] that performs the work of preparing for content installation. The terminal device 10[k] may transmit captured images captured by the imaging device 13 and location information generated by the GPS device 17 to the location information server 30, and the location information server 30 may generate data indicating the spatial structure from the captured images and location information received from the terminal device 10[k], and may generate VPS (Visual Positioning System) information based on the data indicating the spatial structure.

[0015] Furthermore, the terminal device 10[k] includes an imaging device 13, as described below. The terminal device 10[k] transmits a captured image PI captured by the imaging device 13 to the location information server 30. The location information server 30 generates VPS (Visual Positioning System) information based on the captured image PI received from the terminal device 10[k]. Furthermore, the location information server 30 transmits the generated VPS information to the terminal device 10[k] as first location information.

[0016] The content server 50 provides various data and cloud services to the terminal devices 10[1] to 10[j] via the communication network NET. The content server 50 also provides various virtual objects VO to be displayed in the augmented reality space AS to the terminal devices 10[1] to 10[j].

[0017] The terminal device 10[k] displays a virtual object VO placed in an augmented reality space AS on a display device 15 (described later) included in the terminal device 10[k]. The virtual object VO may be, for example, a still image, a video, or a 3DCG model. The terminal device 10[k] is preferably, for example, a mobile terminal device such as a smartphone, smart glasses, or a tablet.

[0018] 1-1-2: Location information server configuration 2 is a block diagram showing an example of the configuration of location information server 30. Location information server 30 includes a processing device 31, a storage device 32, a communication device 33, and an input device 34. The elements of location information server 30 are connected to each other using one or more buses for communicating information.

[0019] The processing device 31 is a processor that controls the entire location information server 30. The processing device 31 is configured, for example, using one or more chips. The processing device 31 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 31 may be realized using hardware such as a DSP, an ASIC, a PLD, or an FPGA. The processing device 31 executes various processes in parallel or sequentially.

[0020] The storage device 32 is a recording medium that can be read from and written to by the processing device 31. The storage device 32 also stores a plurality of programs including a control program PR3 that the processing device 31 executes.

[0021] The storage device 32 also stores a landmark database ODB and a 3D map database MDB that are used when calculating the first position information.

[0022] The landmark database ODB is a database in which landmark data OD, which is data on landmarks, is stored. As described below, a user U[k] captures an image of a landmark using an imaging device 13. In the terminal device 10[k], the imaging device 13 capturing an image of the landmark triggers a distance sensor 19, described below, to generate data indicating the spatial structure of the space in which the landmark exists. The terminal device 10[k] transmits the data indicating the spatial structure to the location information server 30. A first calculation unit 314, described below, included in the location information server 30 extracts feature points from the spatial structure. The first calculation unit 314 compares the data indicating the feature points with feature point data stored in the landmark database ODB, and calculates the position of the terminal device 10[k] in the real space RS and the direction in which the terminal device 10[k] is facing as first location information. In the terminal device 10[k], when the imaging device 13 captures an image of a target, the terminal device 10[k] may send a request to the location information server 30, and the first calculation unit 314 included in the location information server 30 may extract feature points from the spatial structure using data indicating the spatial structure that has already been transmitted from another terminal device 10[x] to the location information server 30. Specifically, when the imaging device 13 captures an image of a target, the terminal device 10[k] may send the captured image captured by the imaging device 13 and the location information generated by the GPS device 17 to the location information server 30. The first calculation unit 314 included in the location information server 30 generates data indicating the spatial structure from the captured image and location information received from the terminal device 10[k], and extracts feature points from the spatial structure. The first calculation unit 314 compares the data indicating the feature points with the feature point data stored in the landmark database ODB, and calculates the position of the terminal device 10[k] in the real space RS and the direction in which the terminal device 10[k] is facing as first position information.

[0023] 3 is a table showing an example of the configuration of the landmark database ODB. Each landmark data OD stored in the landmark database ODB has the following items: "Target ID," "Image data," "Feature point data," and "Location data."

[0024] "Target ID" is an identifier that identifies a target. "Image data" is data that represents an image of a target. "Feature point data" is data indicating feature points of the spatial structure of the space in which the target indicated by the "target ID" exists. "Location data" is data that indicates the latitude and longitude of the location where the target identified by the "target ID" is installed. As an example, the "image data" of the target with the "target ID" of "001" is "img1.jpg". Furthermore, the "feature point data" that indicates the spatial structure of the space where the target with the "target ID" of "001" exists is "dt[1]". Furthermore, the latitude of the location where the target with the "target ID" of "001" is installed is "35.68097698" and the longitude is "139.98163876".

[0025] FIG. 4 illustrates a method for calculating first location information using the landmark database ODB. The location information server 30 transmits an image OI of a landmark indicated by "image data" in the landmark database ODB to the terminal device 10[k]. The server transmitting the image OI to the terminal device 10[k] is not limited to the location information server 30; it may be a server communicatively connected to the terminal device 10[k] via the communication network NET. In this case, the server stores the image OI of the landmark indicated by "image data" in the landmark database ODB and transmits the image OI of the landmark indicated by "image data" in the landmark database ODB to the terminal device 10[k] in response to an explicit or implicit request. The display device 15 of the terminal device 10[k] displays the image OI of the landmark. In the terminal device 10[k], the imaging device 13 captures an image of the landmark and generates a captured image PI, triggering transmission of the captured image PI and location information generated by the GPS device 17 from the terminal device 10[k] to the location information server 30. The location information server 30 generates data indicating a spatial structure from the captured image PI and location information received from the terminal device 10[k], extracts feature points from the spatial structure, and compares the extracted feature points with feature points indicated by the "feature point data" in the landmark database ODB to calculate the first location information. As another example, in the terminal device 10[k], the imaging device 13 may capture an image of a target and generate a captured image PI, triggering the distance sensor 19 (described below) to generate data indicating the spatial structure of the space in which the target exists. In this case, the data indicating the spatial structure may be transmitted from the terminal device 10[k] to the location information server 30, and the location information server 30 may extract feature points from the spatial structure and compare the extracted feature points with feature points indicated by the "feature point data" in the landmark database ODB to calculate the first location information.

[0026] In FIG. 2, the 3D map database MDB is a database of three-dimensional maps. The three-dimensional map may be, for example, a 360-degree panoramic image associated with location information. Another example of the three-dimensional map may be a stereoscopic map. The terminal device 10[k] transmits image data indicating a captured image PI, which is an image of a scene in front of the user U[k] captured by the imaging device 13, to the location information server 30. A second calculation unit 315 (described later) included in the location information server 30 compares features of the captured image PI indicated by the image data with features of an image indicating a three-dimensional map stored in the 3D map database MDB. Based on the comparison result, the second calculation unit 315 calculates the position of the terminal device 10[k] in the real space RS and the direction in which the terminal device 10[k] is facing as first location information. Note that the image indicating the three-dimensional map is an example of a "reference image."

[0027] The communication device 33 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 33 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 33 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 33 may also include a wireless communication interface. Examples of connectors and interface circuits for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of wireless communication interfaces include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0028] The input device 34 is a device that receives operations from the administrator of the position information server 30. For example, the input device 34 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse.

[0029] The processing device 31, for example, reads and executes a control program PR3 from the storage device 32. As a result, the processing device 31 functions as a communication control unit 311, an acquisition unit 312, a determination unit 313, a first calculation unit 314, and a second calculation unit 315.

[0030] The communication control unit 311 causes the communication device 33 to transmit and receive various data to and from devices external to the location information server 30 .

[0031] The acquisition unit 312 acquires, from the terminal device 10[k], data indicating the spatial structure or image data indicating a captured image PI of a scene in front of the user U[k].

[0032] The determination unit 313 determines whether the data acquired by the acquisition unit 312 is data indicating the spatial structure or image data indicating a captured image PI that captures a scene in front of the user U[k].

[0033] When the data acquired by the acquisition unit 312 is determined to be data indicating the spatial structure, the first calculation unit 314 compares feature points of the spatial structure indicated by the data acquired by the acquisition unit 312 with feature points indicated by the "feature point data" included in the landmark data OD stored in the landmark database ODB stored in the storage device 32. When both feature points match at a predetermined rate or more, the first calculation unit 314 determines the position indicated by the "position data" as the position of the terminal device 10[k] in the real space RS. Furthermore, the first calculation unit 314 calculates the orientation of the terminal device 10[k] in the real space RS based on the comparison results. The first calculation unit 314 determines information indicating the position of the terminal device 10[k] in the real space RS and the direction in which the terminal device 10[k] is facing as first position information.

[0034] When the second calculation unit 315 determines that the data acquired by the acquisition unit 312 is image data representing a captured image PI of a scene in front of the user U[k], the second calculation unit 315 compares the image data acquired by the acquisition unit 312 with a three-dimensional map stored in a 3D map database MDB stored in the storage device 32. Based on the comparison result, the second calculation unit 315 calculates the position of the terminal device 10[k] in the real space RS and the direction in which the terminal device 10[k] is facing. More specifically, the second calculation unit 315 extracts multiple feature points from the captured image PI represented by the image data. Furthermore, the second calculation unit 315 compares the extracted feature points with feature points included in the three-dimensional map. Based on the comparison result, the second calculation unit 315 calculates the position of the terminal device 10[k] in the real space RS and the direction in which the terminal device 10[k] is facing as first position information. Note that the image representing the three-dimensional map is an example of a "reference image."

[0035] As will be described later, the GPS information generated by the GPS device 17 included in the terminal device 10[k] is second location information.

[0036] The first location information calculated by the first calculation unit 314 or the second calculation unit 315 is transmitted by the communication control unit 311 to the terminal device 10[k].

[0037] Comparing the calculation method of the first position information by the first calculation unit 314 and the calculation method of the first position information by the second calculation unit 315, the calculation method by the first calculation unit 314 can only calculate first position information that indicates a position included in a narrow area where a target is present, compared to the calculation method by the second calculation unit 315. Furthermore, the calculation method by the first calculation unit 314 can be performed both indoors and outdoors. On the other hand, the calculation method by the second calculation unit 315 can only be performed outdoors.

[0038] 1-1-3: Content server configuration 5 is a block diagram showing an example of the configuration of content server 50. Content server 50 includes a processing device 51, a storage device 52, a communication device 53, and an input device 54. The elements of content server 50 are connected to each other using one or more buses for communicating information.

[0039] The processing device 51 is a processor that controls the entire content server 50. The processing device 51 is configured, for example, using one or more chips. The processing device 51 is configured, for example, using a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 51 may be realized using hardware such as a DSP, an ASIC, a PLD, and an FPGA. The processing device 51 executes various processes in parallel or sequentially.

[0040] The storage device 52 is a recording medium that can be read from and written to by the processing device 51. The storage device 52 also stores a plurality of programs including a control program PR5 that the processing device 51 executes.

[0041] The storage device 52 also stores an object database CDB[1] that stores object data CD1, and a user database UDB that stores user data UD. The object data CD1 is data related to a virtual object VO to be provided to the terminal device 10. The user data UD is data related to a user U of the terminal device 10.

[0042] 6 is a table showing an example of the configuration of the object database CDB[1]. Each object data CD1 stored in the object database CDB[1] has the following fields: "Object ID," "Type flag," "Data file," "Icon file," "Spot position data," "Spot position image file," "Placement position data," and "Number of times usable."

[0043] The "object ID" is an identifier that identifies the virtual object VO. The "type flag" is a flag that indicates the type of the virtual object VO indicated by the "object ID". A virtual object VO whose "type flag" is "1" is placed at a position determined based on the first self-position indicated by the first position information in the augmented reality space AS visually recognized by the user U[k] using the terminal device 10[k]. More specifically, the position at which the virtual object VO whose "type flag" is "1" is placed is determined based on the first self-position indicated by the first position information calculated by the position information server 30 by referring to the landmark database ODB. A virtual object VO whose "type flag" is "2" is placed at a position determined based on the first self-position indicated by the first position information in the augmented reality space AS visually recognized by the user U[k] using the terminal device 10[k]. More specifically, the position at which the virtual object VO whose "type flag" is "2" is placed is determined based on the first self-position indicated by the first position information calculated by the position information server 30 by referring to the 3D map database MDB. A virtual object VO whose "type flag" is "3" is placed at a position determined based on the second self-position indicated by the second position information in the augmented reality space AS viewed by the user U[k] using the terminal device 10[k]. More specifically, the position at which the virtual object VO whose "type flag" is "3" is placed is determined based on the second self-position indicated by the second position information estimated by the terminal device 10 using the GPS device 17 provided in the terminal device 10.

[0044] 7 is a diagram showing an example of a display image DI1 including a virtual object VO1 whose "type flag" is "1." The display image DI1 is an image obtained by superimposing a layout image AI1 in which the virtual object VO1 is laid out on the screen of the display device 15 and a captured image PI captured by the imaging device 13. Specifically, the virtual object VO1 is placed in the augmented reality space AS displayed on the display device 15. In the example shown in Fig. 7, the virtual object VO1 is an advertisement image. The virtual object VO1 is placed on the surface of the wall of a station building that exists in the real space RS included in the augmented reality space AS.

[0045] 8 is a diagram showing an example of a display image DI2 including a virtual object VO2 whose "type flag" is "2." The display image DI2 is an image obtained by superimposing a layout image AI2 in which the virtual object VO2 is laid out on the screen of the display device 15 and a captured image PI captured by the imaging device 13. Specifically, the virtual object VO2 is placed in the augmented reality space AS displayed on the display device 15. In the example shown in Fig. 8, the virtual object VO2 is a three-dimensional image of a panda, and is placed in front of a train station that exists in the real space RS included in the augmented reality space AS.

[0046] 9 is a diagram showing an example of a display image DI3 including a virtual object VO3 whose "type flag" is "3." The display image DI3 is an image obtained by superimposing an arrangement image AI3 in which the virtual object VO3 is arranged on the screen of the display device 15 and a captured image PI captured by the imaging device 13. Specifically, the virtual object VO3 is placed in the augmented reality space AS displayed on the display device 15. In the example shown in Fig. 9, the virtual object VO3 is a cylinder, and is placed on a plane in front of the station building that exists in the real space RS included in the augmented reality space AS.

[0047] In the augmented reality space AS, the accuracy of the position information required to display the virtual object VO varies depending on the type of the virtual object VO. Specifically, for example, when a virtual object VO is displayed superimposed on an object existing in the real space RS, high accuracy is required for the position information required to display the virtual object VO. For example, when the virtual object VO moves, high accuracy is required for the position information required to display the virtual object VO that moves a large amount in the augmented reality space AS. For example, high accuracy is required for the position information required to display a virtual object VO that can only be placed in a small area because of the presence of roads in the surrounding area. On the other hand, low accuracy is sufficient for the position information required to display a virtual object VO that is not superimposed on an object existing in the real space RS, a virtual object VO that does not move, or a virtual object VO that moves a small amount in the augmented reality space AS. For example, low accuracy is sufficient for the position information required to display a virtual object VO that can be placed in a wide area.

[0048] In the above example, the amount of movement of virtual objects VO2 and VO3 in augmented reality space AS is greater than the amount of movement of virtual object VO1. Virtual object VO1 is an advertisement image placed on the surface of the wall of a station building in real space RS, and is therefore a virtual object that does not move. Virtual objects VO2 and VO3 are virtual objects that are placed in front of the station building in real space RS. Furthermore, the range in which virtual object VO1 can be placed is the narrowest, and the range in which virtual object VO3 can be placed is the widest.

[0049] Therefore, the positional accuracy required to display the virtual object VO1 whose "type flag" is "1" is the highest, and the positional accuracy required to display the virtual object VO3 whose "type flag" is "3" is the lowest.

[0050] Therefore, as described above, the position where the virtual object VO1 whose "type flag" is "1" is placed is determined based on the first self-position. The first self-position is determined based on the first position information calculated by the position information server 30 by referring to the landmark database ODB. The position where the virtual object VO2 whose "type flag" is "2" is placed is determined based on the first position information calculated by the position information server 30 by referring to the 3D map database MDB. The position where the virtual object VO3 whose "type flag" is "3" is placed is determined based on the second self-position indicated by the second position information estimated by the terminal device 10 using the GPS device 17.

[0051] The virtual object VO1 whose "type flag" is "1" and the virtual object VO2 whose "type flag" is "2" are examples of the "first virtual object VO[1]". The virtual object VO3 whose "type flag" is "3" is an example of the "second virtual object VO[2]".

[0052] As described above, the movement amounts of the virtual objects VO2 and VO3 are greater than the movement amount of the virtual object VO1. That is, if the first virtual object VO[1] is the virtual object VO1, the second virtual object VO[2] is the virtual object VO3, the first virtual object VO[1] moves within a first range, and the second virtual object VO[2] moves within a second range, the first range is narrower than the second range.

[0053] In FIG. 6, "Data File" is the data file of the virtual object VO indicated by the "Object ID".

[0054] An "icon file" is an image file of an icon indicating a virtual object VO indicated by an "object ID" in an object list CL (described later) displayed on the terminal device 10[k]. The object list CL is a list of virtual objects VO arranged in a predetermined area. Details of the object list CL will be described later with reference to FIG. 15.

[0055] The "spot position data" indicates the latitude and longitude of the spot position. The "spot position" is the center point of the area where the virtual object VO indicated by the "object ID" is placed. As an example, the ground of the real space RS included in the augmented reality space AS is divided into a mesh of multiple areas, each having a square shape with a predetermined distance on each side. Each virtual object VO indicated by the "object ID" is placed in one of the areas. The "spot position data" indicates the latitude and longitude of the center point of the area.

[0056] The "spot position image file" is an image file of an icon indicating the spot position in an object map CM (described later) displayed on the terminal device 10[k]. The object map CM is a map indicating the position of a virtual object VO that exists within a predetermined distance from the terminal device 10[k]. The object map CM is an example of a "map." Details of the object map CM will be described later with reference to FIG. 15.

[0057] Referring to the object database CDB[1] shown in FIG. 6, in the object map CM, the spot position that is the center point of the area where the virtual object VO1 whose "type flag" is "1" is placed is indicated by an icon of the image file "α.gif". Also, the spot position that is the center point of the area where the virtual object VO2 whose "type flag" is "2" is placed is indicated by an icon of the image file "β.gif". Also, the spot position that is the center point of the area where the virtual object VO3 whose "type flag" is "3" is placed is indicated by an icon of the image file "γ.gif". The area where the virtual object VO1 with a "type flag" of "1" is placed and the area where the virtual object VO2 with a "type flag" of "2" is placed are examples of the "first area." The area where the virtual object VO3 with a "type flag" of "3" is placed is an example of the "second area." In the object map CM, the area where the virtual object VO1 whose "type flag" is "1" is placed is indicated by an icon of the image file "α.gif". The area where the virtual object VO2 whose "type flag" is "2" is placed is indicated by an icon of the image file "β.gif". The area where the virtual object VO3 whose "type flag" is "3" is placed is indicated by an icon of the image file "γ.gif". The icon indicated by the image file "α.gif" and the icon indicated by the image file "β.gif" are examples of the "first icon IC[1]". The icon indicated by the image file "γ.gif" is an example of the "second icon IC[2]".

[0058] The "placement position data" indicates the latitude and longitude of the position where the virtual object VO indicated by the "object ID" is placed. More specifically, the "placement position data" indicates the latitude and longitude of the position where the virtual object VO is placed. The position indicated by the "placement position data" is included in the area where the spot position is indicated by the "spot position data." Referring to the object database CDB[1] shown in FIG. 6, the latitude and longitude of the location where the virtual object VO with the "object ID" of "001" is placed are "35.68097696" and "139.98163879." These latitudes and longitudes are included in the area where the latitude of the spot position is "35.68097698" and the longitude is "139.98163876."

[0059] The "usable count" indicates the number of times that the virtual object VO indicated by the "object ID" can be used. Referring to the object database CDB[1] shown in Fig. 6, it is shown that the virtual object VO with the "object ID" of "001" can be used up to five times.

[0060] Referring to the object database CDB[1], the virtual object VO with the "object ID" of "002" and the virtual object VO with the "object ID" of "003" are both located within an area with a spot position latitude of "35.68097685" and a longitude of "139.98163854". In the object map CM, the spot position is indicated by an icon of the image file "γ.gif". Similarly, a virtual object VO with an "object ID" of "004" and a virtual object VO with an "object ID" of "004" are both located within an area with a spot position latitude of "35.68097678" and a longitude of "139.98163865." In the object map CM, the spot position is indicated by an icon of the image file "β.gif."

[0061] 10 is a table showing an example of the configuration of the user database UDB. Each piece of user data UD stored in the user database UDB has the following fields: “User ID,” “Object ID,” “Number of times used,” and “Number of times remaining used.”

[0062] "User ID" is an identifier that identifies user U.

[0063] The "object ID" is an identifier that identifies the virtual object VO used by the user U indicated by the "user ID." The "object ID" is the same identifier as the "object ID" in the object database CDB[1]. As an example, in the user database UDB, an "object ID" of "001" indicates a virtual object VO whose "object ID" in the object database CDB[1] is "001."

[0064] The "number of times used" indicates how many times the user U indicated by the "user ID" can use the virtual object VO indicated by the "object ID." As an example, the user database UDB in Fig. 10 stores data indicating that the user U whose "user ID" is "001" has already used the virtual object VO whose "object ID" is "001" once.

[0065] The "remaining number of uses" indicates how many times the user U indicated by the "user ID" can use the virtual object VO indicated by the "object ID." As an example, the user database UDB in Fig. 10 stores data indicating that the user U whose "user ID" is "001" can use the virtual object VO whose "object ID" is "001" four more times. When the "remaining number of uses" is "0", the virtual object VO whose "remaining number of uses" is "0" is not displayed on the terminal device 10 used by the user U.

[0066] In FIG. 5, the communication device 53 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 53 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 53 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 53 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with cellular communication, wireless LAN, Bluetooth (registered trademark), etc.

[0067] The input device 54 is a device that receives operations from the administrator of the content server 50. For example, the input device 54 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse.

[0068] The processing device 51, for example, reads and executes the control program PR5 from the storage device 52. As a result, the processing device 51 functions as a communication control unit 511, an acquisition unit 512, and an extraction unit 513.

[0069] The communication control unit 511 causes the communication device 53 to transmit and receive various data to and from devices external to the content server 50.

[0070] The acquisition unit 512 acquires, via the communication device 53, the identifier of the user U[k] and the second location information from the terminal device 10[k].

[0071] The extraction unit 513 refers to the object database CDB[1] and extracts object data CD1 corresponding to a virtual object VO whose position indicated by the "spot position data" in the augmented reality space AS is located within a predetermined distance from the position indicated by the second position information.

[0072] Furthermore, the extraction unit 513 refers to the user database UDB and extracts user data UD whose "user ID" is the identifier of the user U[k].

[0073] The object data CD1 and the user data UD are transmitted via the communication device 53 to the terminal device 10[k].

[0074] 1-1-4: Terminal device configuration 11 is a block diagram showing an example configuration of a terminal device 10[k]. The terminal device 10[k] includes a processing device 11, a storage device 12, an imaging device 13, a communication device 14, a display device 15, an input device 16, a GPS device 17, an inertial sensor 18, and a distance sensor 19. The terminal device 10[k] may include the processing device 11, the storage device 12, the imaging device 13, the communication device 14, the display device 15, the input device 16, the GPS device 17, and the inertial sensor 18 without including the distance sensor 19. The elements of the terminal device 10[k] are connected to each other using one or more buses for communicating information.

[0075] The processing device 11 is a processor that controls the entire terminal device 10[k]. The processing device 11 is configured, for example, using one or more chips. The processing device 11 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing device 11 may be realized using hardware such as a DSP, ASIC, PLD, and FPGA. The processing device 11 executes various processes in parallel or sequentially.

[0076] The storage device 12 is a recording medium that can be read from and written to by the processing device 11. The storage device 12 also stores a plurality of programs including a control program PR1 that the processing device 11 executes.

[0077] The storage device 12 also stores an object database CDB[2] that stores object data CD2, and map data MD. The object data CD2 is data related to a virtual object VO that is placed in an augmented reality space AS that is displayed on the display device 15. The map data MD is data that indicates a map that serves as the basis for an object map CM that is displayed on the display device 15.

[0078] 12 is a table showing an example of the configuration of the object database CDB[2]. Each object data CD2 stored in the object database CDB[2] has the following fields: "Object ID," "Type Flag," "Data File," "Icon File," "Spot Position Data," "Spot Position Image File," "Placement Position Data," "Number of Times Used," and "Number of Times Remaining."

[0079] Each piece of object data CD2 is managed by the management unit 112, which will be described later. More specifically, the management unit 112 generates the object data CD2 by merging the object data CD1 and the user data UD received from the content server 50, using the "object ID" as a common element. Specifically, the "object ID," "type flag," "data file," "icon file," "spot position data," "spot position image file," and "placement position data" included in the object data CD2 each match the "object ID," "type flag," "data file," "icon file," "spot position data," "spot position image file," and "placement position data" included in the object data CD1, respectively. Furthermore, the "number of times used" and "number of times remaining used" included in the object data CD2 each match the "number of times used" and "number of times remaining used" included in the user data UD.

[0080] In Fig. 11, the imaging device 13 outputs imaging information obtained by capturing an image of the outside world. The imaging device 13 also includes, for example, a lens, an imaging element, an amplifier, and an AD converter. The imaging element converts light collected through the lens into an imaging signal, which is an analog signal. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is output to the processing device 11.

[0081] The communication device 14 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 14 is also referred to as, for example, a network device, a network controller, a network card, or a communication module. The communication device 14 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 14 may also include a wireless communication interface. Examples of connectors and interface circuits for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of wireless communication interfaces include products that comply with cellular communication via a mobile communication network, wireless LAN, Bluetooth (registered trademark), and the like.

[0082] The display device 15 is a device that displays images and text information. The display device 15 displays various images under the control of the processing device 11. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitably used as the display device 15.

[0083] The input device 16 receives operations from the user U[k]. For example, the input device 16 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 16 includes a touch panel, it may also serve as the display device 15.

[0084] The GPS device 17 receives radio waves from multiple satellites. The GPS device 17 also generates GPS information as second location information from the received radio waves. The GPS information may be in any format as long as it can identify the location. The GPS information indicates, for example, the latitude and longitude of the terminal device 10[k] or the user U[k]. The generated second location information is output to the processing device 11.

[0085] The inertial sensor 18 is a sensor that detects inertial force. The inertial sensor 18 includes, for example, one or more sensors selected from an acceleration sensor, an angular velocity sensor, and a gyro sensor. The processing device 11 detects the attitude of the terminal device 10[k] based on the output information of the inertial sensor 18. Furthermore, the processing device 11 accepts the selection of a virtual object VO, the input of text, and the input of instructions in the augmented reality space AS based on the attitude of the terminal device 10[k]. For example, when the user U[k] operates the input device 16 while pointing the central axis of the terminal device 10[k] toward a predetermined area in the augmented reality space AS, the virtual object VO located in the predetermined area is selected. The operation of the user U[k] on the input device 16 is, for example, a double tap. By operating the terminal device 10[k] in this way, the user U[k] can select a virtual object VO without looking at the input device 16 of the terminal device 10[k].

[0086] The distance sensor 19 is a sensor that measures the distance from the distance sensor 19 to an object located within a predetermined distance from the terminal device 10[k]. The distance sensor 19 is, for example, a LiDAR (Light Detection And Ranging) sensor or a ToF (Time Of Flight) sensor.

[0087] The processing device 11 reads and executes the control program PR1 from the storage device 12. As a result, the processing device 11 functions as a communication control unit 111, a management unit 112, a reception unit 113, a determination unit 114, an operation control unit 115, a detection unit 116, a first generation unit 117, a second generation unit 118, and a display control unit 119.

[0088] The communication control unit 111 causes the communication device 14 to transmit and receive various types of data to and from devices external to the terminal device 10[k].

[0089] The management unit 112 manages virtual objects VO. Referring to the examples shown in FIGS. 7 to 9, the management unit 112 manages virtual objects VO1 to VO3. Of these, virtual objects VO1 to VO2 are, as described above, an example of a "first virtual object VO[1]." The virtual object VO3 is, as described above, an example of a "second virtual object VO[2]."

[0090] More specifically, as described above, the management unit 112 manages each object data CD2 stored in the object database CDB[2]. In managing the object data CD2, the management unit 112 manages the "data file" included in the object data CD2. As described above, the "data file" is the data file of the virtual object VO indicated by the "object ID." In other words, the management unit 112 manages the data file of the virtual object VO.

[0091] The management unit 112 also manages whether or not the virtual object VO is displayed on the display device 15. In particular, the management unit 112 manages whether or not the first virtual object VO[1] and the second virtual object VO[2] are displayed on the display device 15.

[0092] Specifically, the management unit 112 manages the data of "used count" included in each object data CD2 as the number of times the virtual object VO has been used. If the virtual object VO has been used even once, the management unit 112 considers the virtual object VO to be displayed on the display device 15. On the other hand, if the virtual object VO has never been used, the management unit 112 considers the virtual object VO to be not displayed on the display device 15.

[0093] The management unit 112 also manages the areas in which the virtual objects VO are placed. In particular, the management unit 112 manages a first area in which the first virtual object VO[1] is placed and a second area in which the second virtual object VO[2] is placed.

[0094] As described above, the ground surface of the real space RS included in the augmented reality space AS is divided into a mesh of areas, each of which has a square shape with a predetermined length on each side, and each virtual object VO is placed in one of the areas. The management unit 112 manages the areas.

[0095] The reception unit 113 receives an operation of the user U[k]. More specifically, the reception unit 113 mainly receives an operation of the user U[k] on the input device 16.

[0096] The determination unit 114 determines the own position of the terminal device 10[k] based on each of the first position information received from the position information server 30 and the second position information generated by the GPS device 17. The own position of the terminal device 10[k] determined based on the first position information is an example of a “first own position.” The own position of the terminal device 10[k] determined based on the second position information is an example of a “second own position.” Furthermore, the first method of determining self-location based on the first position information is an example of a “first self-location determination method.” The second method of determining self-location based on the second position information is an example of a “second self-location determination method.” As described above, the second self-location determination method has lower determination accuracy than the first self-location determination method.

[0097] 13 is a functional block diagram of the determination unit 114. The estimation unit 114 includes an acquisition unit 114[1] and an estimation unit 114[2].

[0098] The acquisition unit 114[1] corresponds to the first self-location determination method. As described above, the first self-location determination method is based on the first location information received from the location information server 30. That is, the first self-location determination method includes a method of acquiring, as the first self-location, first location information calculated by the location information server 30 based on feature points of the spatial structure of the space where the target exists (which may be feature points of the spatial structure of the space where the target exists detected by the distance sensor 19) and feature points stored in the target database ODB of the location information server 30. The acquisition unit 114[1] acquires the first self-location when a target exists within the imaging range of the captured image PI. In addition, the first self-position determination method includes a method of acquiring, as the first self-position, first position information calculated by the position information server 30 based on the characteristics of the captured image PI captured by the imaging device 13 and the characteristics of a reference image prepared in advance.

[0099] As described above, the first self-location determination method is not executed all the time because it requires the imaging device 13 to capture an image of the target or the scenery in the vicinity of the terminal device 10[k].

[0100] The estimation unit 114[2] corresponds to a second self-location determination method. The second self-location determination method is based on GPS information as second position information generated by the GPS device 17. That is, the second self-location determination method is a method of estimating the second self-location based on radio waves received from multiple satellites. In addition, the second self-location determination method is, for example, constantly executed intermittently.

[0101] The operation control unit 115 operates the imaging device 13 and the acquisition unit 114[1] when the second self-position estimated by the estimation unit 114[2] is located within the first area. More specifically, when the second self-position is located within the first area, the imaging device 13 and the acquisition unit 114[1] are ready to operate. Based on an operation from the user U[k] accepted by the acceptance unit 113, the imaging device 13 captures an image of a target or a scene in front of the user U[k], which triggers the acquisition unit 114[1] to acquire the first self-position.

[0102] The detection unit 116 detects the presence or absence of a substance contained in the ground in the second area based on the image captured by the imaging device 13. Based on the measurement values ​​measured by the distance sensor 19, a plane included in the ground in the second area may be detected.

[0103] The first generation unit 117 generates an arrangement image AI1 in which the virtual object VO1 is arranged on the screen of the display device 15 based on the first self-position. More specifically, the first generation unit 117 generates an arrangement image AI1 in which the virtual object VO1 is arranged at a position indicated by the "arrangement position data" of the object data CD2, based on the first self-position, in the augmented reality space AS displayed on the display device 15. Furthermore, the first generation unit 117 generates a display image DI1 in which the arrangement image AI1 is superimposed on a captured image PI captured by the imaging device 13. An example of the display image DI1 is shown in FIG. 7 above.

[0104] Similarly, the first generation unit 117 generates an arrangement image AI2 in which the virtual object VO2 is arranged on the screen of the display device 15 based on the first self-position. More specifically, the first generation unit 117 generates an arrangement image AI2 in which the virtual object VO2 is arranged at a position indicated by the "arrangement position data" of the object data CD2, based on the first self-position, in the augmented reality space AS displayed on the display device 15. Furthermore, the first generation unit 117 generates a display image DI2 in which the arrangement image AI2 is superimposed on a captured image PI captured by the imaging device 13. An example of the display image DI2 is shown in FIG. 8 above. The layout image AI1 and the layout image AI2 are examples of a “first layout image AI[1].” Furthermore, the display image DI1 and the display image DI2 are examples of a “first display image DI[1].”

[0105] Furthermore, the first generation unit 117 generates an arrangement image AI3 in which the virtual object VO3 is arranged on the screen of the display device 15 based on the second self-position. More specifically, the first generation unit 117 generates an arrangement image AI3 in which the virtual object VO3 is arranged on a plane detected by the detection unit 116 in the augmented reality space AS displayed on the display device 15. The plane includes the position indicated by the "arrangement position data" of the object data CD2 based on the second self-position. Note that the plane may be a plane near the position indicated by the "arrangement position data" of the object data CD2. Furthermore, the first generation unit 117 generates a display image DI3 in which the arrangement image AI3 is superimposed on a captured image PI captured by the imaging device 13. An example of the display image DI3 is shown in FIG. 9 above. The layout image AI3 is an example of a "second layout image AI[2]." The display image DI3 is an example of a "second display image DI[2]."

[0106] In FIG. 11, the second generating unit 118 generates an object map CM.

[0107] FIG. 14 is a diagram showing an example of an object map CM. The object map CM shown in FIG. 14 displays concentric rings C1 and C2. The ring C1 is located inside the ring C2. The ring C1 indicates the second self-location of the terminal device 10[k]. The ring C2 indicates a range within a predetermined distance from the second self-location. The predetermined distance is the same as the predetermined distance used as a reference when the extraction unit 513 included in the content server 50 extracts the object data CD1. In addition, an indication line DL is added to the ring C1. The indication line DL indicates the direction in which the user U[k] holding the terminal device 10[k] is facing. The direction in which the user U[k] is facing is calculated, for example, based on measurements taken by at least one of the inertial sensor 18 and a geomagnetic sensor (not shown).

[0108] Icons IC1 to IC3 are arranged inside the ring C2. These icons IC1 to IC3 indicate positions in the object map CM that are indicated by the "spot position data" of the object data CD2 stored in the object database CDB[2].

[0109] More specifically, icon IC1 indicates the position indicated by the "spot position data" when the "type flag" included in object data CD2 is "1." Icon IC2 indicates the position indicated by the "spot position data" when the "type flag" included in object data CD2 is "2." Icon IC3 indicates the position indicated by the "spot position data" when the "type flag" included in object data CD2 is "3." Referring to the object database CDB[2] illustrated in FIG. 12, icon IC1 corresponds to the image indicated by "α.gif," which is the "spot position image file." Icon IC2 corresponds to the image indicated by "β.gif," which is the "spot position image file." Icon IC3 corresponds to the image indicated by "γ.gif," which is the "spot position image file."

[0110] Icons IC1 and IC2 are examples of a "first icon IC[1]." Icon IC3 is an example of a "second icon IC[2]." As described above, when the "type flag" included in the object data CD2 is "1" or "2", the "spot position data" indicates the position of the center point of the first area. Also, when the "type flag" included in the object data CD2 is "3", the "spot position data" indicates the position of the center point of the second area. Therefore, the first icon IC[1] indicates the first area. The second icon IC[2] indicates the second area.

[0111] When the user U[k] touches any of the icons IC1 to IC3 in the object map CM on the display device 15, an object list CL is displayed on the display device 15. The object list CL is a list of virtual objects VO arranged in an area whose center point is the position of the icon IC1 to IC3 touched by the user U[k].

[0112] Fig. 15 is a diagram showing an example of the object list CL. The object list CL shown in Fig. 15 is the object list CL that is displayed when the user U[k] touches the icon IC2.

[0113] The object list CL includes one or more boxes BX corresponding to each virtual object VO. Each box BX includes an object icon CI indicating the virtual object VO, an object name NM, and marks MK in the same number as the remaining number of uses. The object icon CI is an image indicated by an "icon file" included in the object data CD2. The number of marks MK matches the "remaining number of uses" included in the object data CD2.

[0114] In the object list CL, one or more boxes BX may be arranged in descending order of importance of the virtual object VO to which the box BX corresponds. As an example, the greater the number of marks MK, i.e., the "remaining number of uses," the higher the importance. Alternatively, the fewer the number of marks MK, i.e., the "remaining number of uses," the higher the importance. Furthermore, when there are multiple virtual objects VO with the same importance, the boxes BX that correspond one-to-one to these virtual objects VO may be arranged in ascending order of the "object ID" included in the object data CD2, or alternatively, these boxes BX may be arranged in descending order of the "object ID" included in the object data CD2.

[0115] The object list CL also displays a message MG prompting the user to start up the imaging device 13, a first button OB for accepting the start-up, and a second button NB for canceling the start-up. When the user U[k] touches the first button OB, the imaging device 13 starts up. When the user U[k] touches the second button NB, the imaging device 13 does not start up.

[0116] In FIG. 11, the display control unit 119 causes the display device 15 to display a display image DI1, a display image DI2, or a display image DI3.

[0117] Furthermore, when the reception unit 113 receives an operation of the user U[k] to display a map, the display control unit 119 causes the display device 15 to display an object map CM exemplified in Fig. 14. As described above, the object map CM includes icons IC1 and IC2, which are the first icon IC[1], and icon IC3, which is the second icon IC[2].

[0118] When the first icon IC[1] and the second icon IC[2] overlap in the object map CM, the display control unit 119 may display the first icon IC[1] and hide the second icon IC[2]. Alternatively, when the first icon IC[1] and the second icon IC[2] overlap in the object map CM, the display control unit 119 may hide the first icon IC[1] and display the second icon IC[2].

[0119] Similarly, when the first icon IC[1] and the second icon IC[2] overlap, the first virtual object VO[1] has already been displayed on the display device 15, and the second virtual object VO[2] has not yet been displayed on the display device 15, the display control unit 119 hides the first icon IC[1] and displays the second icon IC[2]. Conversely, when the first icon IC[1] and the second icon IC[2] overlap, the second virtual object VO[2] has already been displayed on the display device 15, and the first virtual object VO[1] has not yet been displayed on the display device 15, the display control unit 119 displays the first icon IC[1] and hides the second icon IC[2]. Alternatively, the display control unit 119 may not completely hide the icon to be hidden, but may instead change the color of the icon to be hidden, for example, by graying it out (so-called graying out), making it difficult to see compared to other icons. In this way, the display control unit 119 may preferentially display other icons so that the user U can check them preferentially.

[0120] Furthermore, the display control unit 119 causes the display device 15 to display an object list CL as shown in FIG.

[0121] If the object list CL is the object list CL displayed by touching the icon IC1, when the user U[k] touches the first button OB, the acquisition unit 114[1] is activated in addition to the imaging device 13. Furthermore, the communication control unit 111 causes the communication device 14 to receive an image file indicated by the "image data" of the landmark data OD whose "position data" is closest to the second self-position of the terminal device 10[k] in the landmark database ODB of the position information server 30. Furthermore, the communication control unit 111 causes the image file received by the communication device 14 to be stored in the storage device 12.

[0122] The display control unit 119 causes the display device 15 to display an image OI of the target object indicated by the image file, as exemplified in FIG. 4. The user U[k] captures an image of the target object indicated by the image OI of the target object using the imaging device 13. Triggered by the user U[k] capturing an image of the target object, the acquisition unit 114[1] acquires a first self-position. The first generation unit 117 generates a display image DI1 based on the first self-position. The display control unit 119 causes the display device 15 to display the display image DI1.

[0123] If the object list CL is the object list CL displayed by touching the icon IC2, when the user U[k] starts the imaging device 13 by touching the first button OB, the user U[k] captures an image of the scenery in front of the user U[k] with the imaging device 13. Triggered by the user U[k] capturing an image of the scenery in front of the user U[k], the acquisition unit 114[1] acquires a first self-position. The first generation unit 117 generates a display image DI2 based on the first self-position. The display control unit 119 causes the display device 15 to display the display image DI2.

[0124] If the object list CL is the object list CL displayed by touching the icon IC3, when the user U[k] activates the imaging device 13 by touching the first button OB, the second generation unit 118 generates a display image DI3. The display control unit 119 causes the display device 15 to display the display image DI3.

[0125] 1-2: Operation of the first embodiment Fig. 16 is a flowchart showing an example of the operation of the terminal device 10[k]. Specifically, Fig. 16 is a flowchart showing an example of the operation of the terminal device 10[k] to display the first display image DI[1] or the second display image DI[2] on the display device 15.

[0126] In step S1, the processing device 11 functions as the estimation unit 114[2]. The processing device 11 estimates a second self-position based on radio waves received from a plurality of satellites.

[0127] In step S2, the processing device 11 determines whether the second self-location estimated in step S1 is located within the first area. If the second self-location is located within the first area (YES in step S2), the processing device 11 executes the operation of step S3. If the second self-location is not located within the first area (NO in step S2), the processing device 11 executes the operation of step S8.

[0128] In step S3, the processing device 11 determines whether or not it has received an operation by the user U[k] to touch the first button OB. If the processing device 11 has received the operation (YES in step S3), the processing device 11 executes the operation of step S4. If the processing device 11 has not received the operation (NO in step S3), the processing device 11 executes the operation of step S3. Although not shown in FIG. 16, if the processing device 11 has received an operation by the user U[k] to touch the second button NB, the processing device 11 executes the operation of step S1.

[0129] In step S4, the processing device 11 functions as the operation control unit 115. The processing device 11 operates the imaging device 13 and the acquisition unit 114[1].

[0130] In step S5, the processing device 11 functions as the acquisition unit 114[1]. The processing device 11 acquires a first self-location using a first self-location determination method.

[0131] In step S6, the processing device 11 functions as the first generation unit 117. The processing device 11 generates a first arrangement image AI[1] based on the first self-position. Furthermore, the processing device 11 generates a first display image DI[1] by superimposing the first arrangement image AI[1] on a captured image PI captured by the imaging device 13.

[0132] In step S7, the processing device 11 functions as the display control unit 119. The processing device 11 causes the display device 15 to display the first display image DI[1]. Thereafter, the processing device 11 executes the operation of step S1.

[0133] In step S8, the processing device 11 determines whether the second self-location estimated in step S1 is located within the second area. If the second self-location is located within the second area (YES in step S8), the processing device 11 executes the operation of step S9. If the second self-location is not located within the second area (NO in step S8), the processing device 11 executes the operation of step S1.

[0134] In step S9, the processing device 11 functions as the detection unit 116. The processing device 11 detects a plane included in the ground in the second area.

[0135] In step S10, the processing device 11 functions as the second generation unit 118. The processing device 11 generates a second arrangement image AI[2] based on the second self-position. Furthermore, the processing device 11 generates a second display image DI[2] by superimposing the second arrangement image AI[2] on a captured image PI captured by the imaging device 13.

[0136] In step S11, the processing device 11 functions as the display control unit 119. The processing device 11 causes the display device 15 to display the second display image DI[2]. Thereafter, the processing device 11 executes the operation of step S1.

[0137] 1-3: Effects of the First Embodiment The terminal device 10 according to this embodiment includes an estimation unit 114, a management unit 112, and a display control unit 119. The determination unit 114 determines a first self-location using a first self-location determination method, and determines a second self-location using a second self-location determination method that has lower accuracy than the first self-location determination method. The management unit 112 manages a first virtual object VO[1] and a second virtual object VO[2] that can be placed with lower accuracy than the first virtual object VO[1]. The display control unit 119 causes the display device 15 to display a first display image DI[1] in which a first placement image AI[1] in which the first virtual object VO[1] is placed at a position determined based on the first self-location is superimposed on a captured image PI captured by the imaging device 13. In addition, the display control unit 119 causes the display device 15 to display a second display image DI[2] in which a second placement image AI[2] in which a second virtual object VO[2] is placed at a position determined based on the second self-position is superimposed on an imaged image PI captured by the imaging device 13.

[0138] The terminal device 10 has the above configuration, and is therefore capable of displaying an image of the virtual object VO while taking into consideration a plurality of self-position determining methods.

[0139] More specifically, the terminal device 10 can output different display images DI in which different virtual objects VO are arranged, depending on the accuracy of its own position.

[0140] In addition, in the terminal device 10, the determination unit 114 includes an acquisition unit 114[1] and an estimation unit 114[2]. The acquisition unit 114[1] corresponds to a first self-location determination method and acquires a first self-location based on features of the captured image PI and features of a reference image prepared in advance. The estimation unit 114[2] corresponds to a second self-location determination method and estimates a second self-location based on radio waves received from multiple satellites.

[0141] Because the terminal device 10 has the above-mentioned configuration, it is possible to output display images DI in which different virtual objects VO are placed, depending on the accuracy of each of the first self-position determined based on the characteristics of the captured image PI and the characteristics of a pre-prepared reference image, and the second self-position estimated based on radio waves received from multiple satellites.

[0142] Furthermore, in the terminal device 10, the determination unit 114 includes an acquisition unit 114[1] and an estimation unit 114[2]. The acquisition unit 114[1] corresponds to a first self-location determination method and acquires a first self-location identified based on a spatial structure of a space including a target measured by the terminal device 10 and a predetermined spatial structure. The estimation unit 114[2] corresponds to a second self-location determination method and estimates a second self-location based on radio waves received from multiple satellites.

[0143] Since the terminal device 10 has the above configuration, it is possible to output display images DI in which different virtual objects VO are placed, depending on the accuracy of each of the spatial structure of the space including the target measured by the terminal device 10 and the predetermined spatial structure, the first self-position determined based on the spatial structure, and the second self-position estimated based on radio waves received from multiple satellites.

[0144] In addition, in the terminal device 10, the management unit 112 manages a first area in which the first virtual object VO[1] is placed and a second area in which the second virtual object VO[2] is placed. The terminal device 10 further includes an operation control unit 115. When the second self-position estimated by the estimation unit 114[2] is located within the first area, the operation control unit 115 operates the imaging device 13 and the acquisition unit 114[1].

[0145] The terminal device 10 has the above configuration, and therefore can start the operation of the acquisition unit 114[1] based on the second self-location. Therefore, the terminal device 10 does not need to constantly execute the first self-location determination method corresponding to the acquisition unit 114[1].

[0146] In addition, in the terminal device 10, the management unit 112 manages a first area in which a first virtual object VO[1] is placed and a second area in which a second virtual object VO[2] is placed. The terminal device 10 further includes an operation control unit 115. When the second self-position estimated by the estimation unit 114[2] is located within the first area, the operation control unit 115 operates the imaging device 13 and the acquisition unit 114[1]. A target is located within the first area. When the operation control unit 115 operates the imaging device 13, the display control unit 119 displays an image OI of the target stored in the storage device 12 on the display device 15. When the target is present within the imaging range of the captured image PI, the acquisition unit 114[1] estimates the first self-position.

[0147] The terminal device 10 has the above configuration, and therefore, when a target object exists within the imaging range of the captured image PI, it is possible to use a more accurate first self-location.

[0148] Furthermore, in the terminal device 10, the first virtual object VO[1] moves within a first range, and the second virtual object VO[2] moves within a second range, which is narrower than the second range.

[0149] The terminal device 10 has the above configuration, and therefore, when providing a display image DI in which a virtual object VO with a wide range of movement is placed, it is possible to use a more accurate first self-position.

[0150] The terminal device 10 also includes a reception unit 113. The reception unit 113 receives an operation from the user U. When the reception unit 113 receives an operation from the user U to display a map, the display control unit 119 causes the display device 15 to display a map including a first icon IC[1] indicating the first area and a second icon IC[2] indicating the second area.

[0151] Since the terminal device 10 has the above configuration, the user U can recognize areas on the map where the user's own position is more accurately determined.

[0152] In addition, in the terminal device 10, when the first icon IC[1] and the second icon IC[2] overlap, the display control unit 119 displays the first icon IC[1] on the map and hides or grays out the second icon IC[2].

[0153] Since the terminal device 10 has the above configuration, the user U can preferentially check areas where the accuracy of the user's own position is high.

[0154] Furthermore, in the terminal device 10, the management unit 112 manages whether the first virtual object VO[1] and the second virtual object VO[2] are displayed on the display device 15. When a first icon IC[1] indicating a first area in which the first virtual object VO[1] displayed on the display device 15 is located and a second icon IC[2] indicating a second area in which the second virtual object VO[2] not displayed on the display device 15 is located overlap on the map, the display control unit 119 hides the first icon IC[1] and displays the second icon IC[2]. Alternatively, the display control unit 119 may not completely hide the icon to be hidden, but may instead change the color of the icon to be hidden, for example, by graying it out (graying out) to make it difficult to see compared to other icons. In this way, the display control unit 119 may prioritize displaying the other icons so that the user U can check them preferentially.

[0155] Since the terminal device 10 has the above configuration, the user U can preferentially check the area where the unexperienced virtual object VO is placed. Alternatively, the display control unit 119 may change the color of the icon to be hidden, for example, by displaying it in gray (graying out), rather than completely hiding the icon. In this way, the display control unit 119 may preferentially display the other icons, allowing the user U to preferentially check them.

[0156] Furthermore, in the terminal device 10, the management unit 112 manages whether the first virtual object VO[1] and the second virtual object VO[2] are displayed on the display device 15. When a first icon IC[1] indicating a first area in which the first virtual object VO[1] not displayed on the display device 15 is located and a second icon IC[2] indicating a second area in which the second virtual object VO[2] displayed on the display device 15 is located overlap on the map, the display control unit 119 displays the first icon IC[1] and hides the second icon IC[2]. Alternatively, the display control unit 119 may not completely hide the icon to be hidden, but may instead change the color of the icon to be hidden, for example, by graying it out (graying out) to make it difficult to see compared to other icons. In this way, the display control unit 119 may prioritize displaying the other icons so that the user U can check them preferentially.

[0157] The terminal device 10 has the above-described configuration, so that the user U can preferentially check the area where the unexperienced virtual object VO is placed.

[0158] 2: Variation The present disclosure is not limited to the above-described exemplary embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected from the following examples may be combined.

[0159] 2-1: Variation 1 In the above embodiment, the terminal device 10[j] includes a display device 15. The display device 15 may be an external device of the terminal device 10[j]. As an example, the display device 15 may be AR glasses.

[0160] 2-2: Variation 2 In the above embodiment, the terminal device 10[j] provides the user U[j] with a virtual object VO placed in an augmented reality space AS. However, the terminal device 10[j] may also provide the user U[j] with a virtual object VO placed in a mixed reality space. Furthermore, these virtual objects VO may be virtual objects VO used in games.

[0161] 2-3: Variation 3 In the above embodiment, the object map CM includes icons IC1 to IC3 indicating the center points of each area. However, instead of icons IC1 to IC3, the object map CM may use an image indicated by an "icon file" corresponding to the virtual object VO with the highest priority in each area.

[0162] 2-4: Variation 4 The location information server 30 according to the above embodiment stores an object database ODB and a 3D map database MDB. The location information server 30 also includes both a first calculation unit 314 and a second calculation unit 315. However, the set of the object database ODB and the first calculation unit 314 and the set of the 3D map database MDB and the second calculation unit 315 may be provided in a first location information server 30[1] and a second location information server 30[2], which are separate entities.

[0163] 2-5: Variation 5 In the object map CM, when specific icons IC1 to IC3 are outside the display range, the display control unit 119 may automatically adjust the scale of the object map CM so that the specific icons IC1 to IC3 are displayed.

[0164] 2-6: Variation 6 In the object map CM, when a specific icon IC1 to icon IC3 is outside the display range, the display control unit 119 may display within the object map CM that the specific icon IC1 to icon IC3 is not displayed.

[0165] 2-7: Variation 7 When the period during which the virtual object VO placed in the area can be experienced falls below a predetermined number of days, the display control unit 119 may blink the icons IC1 to IC3 placed at the center of the area.

[0166] 2-8: Variation 8 In the above embodiment, the extraction unit 513 included in the content server 50 may change the object data CD2 extracted from the object data CD1 stored in the object database CDB[1] depending on the user U. As an example, the information processing system 1 may provide a specific virtual object VO only to the user U who satisfies a specific condition.

[0167] 3:Other (1) In the above-described embodiment, storage device 12, storage device 32, and storage device 52 are exemplified by ROM and RAM, but may be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium. The program may also be transmitted from a network via a telecommunications line. The program may also be transmitted from a communications network NET via a telecommunications line.

[0168] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0169] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0170] (4) In the above-described embodiment, the determination may be made by a value (0 or 1) represented using one bit, by a Boolean value (true or false), or by a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0171] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0172] (6) Each function illustrated in Figures 1 to 16 is realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, by wire, wirelessly, etc.) and these multiple devices. A functional block may also be realized by combining software with the single device or the multiple devices.

[0173] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0174] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0175] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0176] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0177] (10) In the above-described embodiments, terminal device 10[1] through terminal device 10[j] may be a mobile station (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate term. In addition, in this disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0178] (11) In the above-described embodiments, the terms "connected," "coupled," or any variations thereof refer to any direct or indirect connection or coupling between two or more elements, including the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements are considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0179] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0180] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judgment" or "decision." In other words, "judgment" and "decision" can include regarding some action as having been "judgment" or "decision." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0181] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0182] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are plural.

[0183] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0184] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0185] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0186] 1: Information processing system, IC1: Icon, IC2: Icon, IC3: Icon, 10: Terminal device, 11: Processing device, 12: Storage device, 13: Imaging device, 14: Communication device, 15: Display device, 16: Input device, 17: GPS device, 18: Inertial sensor, 19: Distance sensor, 30: Location information server, 31: Processing device, 32: Storage device, 33: Communication device, 34: Input device, 50: Content server, 51: Processing device, 52: Storage device, 53: Communication device, 54: Input device, 111: Communication control unit, 112: Management unit, 113 : Reception unit, 114: Determination unit, 114[1]: Acquisition unit, 114[2]: Estimation unit, 115: Operation control unit, 116: Detection unit, 117: First generation unit, 118: Second generation unit, 119: Display control unit, 311: Communication control unit, 312: Acquisition unit, 313: Discrimination unit, 314: First calculation unit, 315: Second calculation unit, 511: Communication control unit, 512: Acquisition unit, 513: Extraction unit, AI[1]: First arrangement image, AI[2]: Second arrangement image, AI1: Arrangement image, AI2: Arrangement image, AI3: Arrangement image, AS: Augmented reality space, BX: Box, C1: Circle Ring, C2: Ring, CD1: Object data, CD2: Object data, CDB[1]: Object database, CDB[2]: Object database, CI: Object icon, CL: Object list, CM: Object map, DI: Display image, DI[1]: First display image, DI[2]: Second display image, DI1: Display image, DI2: Display image, DI3: Display image, DL: Indicator line, MD: Map data, MDB: 3D map database, MG: Message, MK: Mark, NB: Second Button, NET: communication network, NM: object name, OB: first button, OD: target data, ODB: target database, OI: image, PI: captured image, PR1: control program, PR3: control program, PR5: control program, RS: real space, U: user, UD: user data, UDB: user database, VO: virtual object, VO[1]: first virtual object, VO[2]: second virtual object, VO1: virtual object, VO2: virtual object, VO3: virtual object

Claims

1. a determination unit that determines a first self-location by a first self-location determination method and determines a second self-location by a second self-location determination method having lower accuracy than the first self-location determination method; a management unit that manages a first virtual object and a second virtual object that can be placed with lower accuracy than the first virtual object; a display control unit that causes a display device to display a first display image in which a first arrangement image, in which the first virtual object is arranged at a position determined based on the first self-position, is superimposed on a captured image captured by an imaging device, and causes the display device to display a second display image in which a second arrangement image, in which the second virtual object is arranged at a position determined based on the second self-position, is superimposed on the captured image; A terminal device comprising:

2. The determination unit an acquisition unit that corresponds to the first self-location determination method and acquires the first self-location identified based on features of the captured image and features of a reference image prepared in advance; an estimation unit that corresponds to the second self-position determination method and estimates the second self-position based on radio waves received from a plurality of satellites, The terminal device according to claim 1 .

3. The determination unit an acquisition unit that corresponds to the first self-location determination method and acquires the first self-location identified based on a spatial structure of a space including a target measured by the terminal device and a predetermined spatial structure; an estimation unit that corresponds to the second self-position determination method and estimates the second self-position based on radio waves received from a plurality of satellites, The terminal device according to claim 1 .

4. the management unit manages a first area in which the first virtual object is placed and a second area in which the second virtual object is placed; an operation control unit that operates the imaging device and the acquisition unit when the second self-position estimated by the estimation unit is located within the first area; The terminal device according to claim 2 .

5. the management unit manages a first area in which the first virtual object is placed and a second area in which the second virtual object is placed; an operation control unit that operates the imaging device and the acquisition unit when the second self-position estimated by the estimation unit is located within the first area; The target is located within the first area, the display control unit causes the display device to display an image of the target object stored in a storage device when the operation control unit operates the imaging device; the acquisition unit acquires the first self-position when the target is present within an imaging range of the captured image. The terminal device according to claim 3.

6. the first virtual object moves within a first range; the second virtual object moves within a second range; The first range is narrower than the second range. The terminal device according to claim 1 .

7. a reception unit that receives a user's operation, the management unit manages a first area in which the first virtual object is placed and a second area in which the second virtual object is placed; The display control unit when the accepting unit accepts an operation by the user to display a map, the map including a first icon indicating the first area and a second icon indicating the second area is displayed on the display device; The terminal device according to claim 1 .

8. When the first icon and the second icon overlap, the display control unit displays the first icon on the map and hides the second icon, or displays the first icon with priority over the second icon. The terminal device according to claim 7.

9. the management unit manages whether the first virtual object and the second virtual object are displayed on the display device; The display control unit When the first icon indicating a first area in which the first virtual object displayed on the display device is to be placed and a second icon indicating a second area in which the second virtual object not displayed on the display device is to be placed overlap on the map, the first icon is hidden and the second icon is displayed, or the second icon is displayed with priority over the first icon. The terminal device according to claim 7.

10. the management unit manages whether the first virtual object and the second virtual object are displayed on the display device; The display control unit When the first icon indicating a first area in which the first virtual object not displayed on the display device is to be placed and the second icon indicating a second area in which the second virtual object displayed on the display device is to be placed overlap on the map, the first icon is displayed and the second icon is not displayed, or the first icon is displayed in priority to the second icon. The terminal device according to claim 7.

Images