Portable information terminal and information processing system

Abstract

To provide a portable information terminal which has a communication function such as a head-mounted display (HMD) with sufficient accuracy without the need of adding a special facility (hardware) and to provide an information processing system that specifies an existence region thereof.SOLUTION: In an existence region specification system 100, a portable information terminal such as an HMD 110 acquires relay device identification information from access points (AP 310) which are multiple relay devices located in a space where the portable information terminal exists, selects the relay device based on the user's instruction or the order of priority stored in a memory, connects to the relay device using a password stored in association therewith, requests spatial shape information from a server via the relay device when the spatial shape information that maps the space required to display on the display is not registered in the space, receives the spatial shape information transmitted from the server, and stores the spatial shape information in association with the relay device identification information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a technique for specifying the presence area of a portable information terminal such as a head-mounted display (hereinafter referred to as "HMD").

Background Art

[0002] In an HMD, a real-space video and a generated image of augmented reality (AR) by a computer (AR image such as an avatar) are superimposed and displayed on a display screen in the form of glasses. Generally, AR images are stored in an external server. In order to accurately extract information regarding the area where the HMD currently exists from a large number of AR images stored in the external server, or in order for the external server to appropriately provide information to each HMD, it is necessary to specify the presence area of the HMD.

[0003] As a technique for specifying the presence area of an HMD, for example, there is the technique described in Patent Document 1. Patent Document 1 describes, "An experience facility is composed of a plurality of divided areas. A detection antenna D receives an identification code generated by a head-mounted display device (HMD) to detect in which divided area among the plurality of divided areas the HMD exists, and to detect in which direction the HMD is facing within the detected divided area. Storage means stores different image information for each of the plurality of divided areas, and extraction means extracts from the image information of the corresponding divided area a portion corresponding to the detected direction from the storage means. The extracted virtual image is displayed on the HMD (summary excerpt).".

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technology described in Patent Document 1, an identification signal is transmitted from the HMD, received by 16 receiving antennas, and the presence area of the HMD is specified based on the relative phases at the respective receiving antennas. Also, different position codes are transmitted from four pre-installed transmitting antennas, and the HMD specifies the presence area from the relative phases of the four received position codes. Therefore, in order to specify the presence area of the HMD, installation of new equipment such as a plurality of antennas is necessary, which incurs a great cost.

[0006] The present invention has been made in view of the above circumstances, and an object thereof is to provide a technology capable of specifying the presence area of a portable information terminal having a communication function such as an HMD with necessary and sufficient accuracy without adding special equipment (hardware).

Means for Solving the Problems

[0007] In order to solve the above problems, the present invention provides a portable information terminal, comprising a display, a communication interface connected to a server via a relay device, an input unit for inputting an instruction of a user of the portable information terminal, a memory, and a processor. The processor acquires relay device identification information, which is information for identifying the plurality of relay devices, from the plurality of relay devices arranged in the space where the portable information terminal exists, selects a relay device to be used for connecting to the server from the plurality of relay devices based on the user instruction input by the input unit or the priority stored in the memory, reads out a password stored in the memory in association with the selected relay device, connects to the selected relay device using the read password, and if space shape information obtained by mapping a space necessary for display on the display in the space is not registered in the memory, requests the server for the space shape information via the selected relay device by the communication interface, receives the space shape information transmitted from the server in response to the request, and stores the received space shape information in the memory in association with the relay device identification information of the selected relay device. [Effect of the Invention]

[0008] The presence area of the HMD can be specified with sufficient accuracy without adding special equipment (hardware). Other objects, configurations, and effects will be clarified in the following embodiments. [Brief Description of the Drawings]

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The same components are denoted by the same reference numerals throughout the drawings, and redundant descriptions are omitted.

[0011] <<First Embodiment>> First, the outline of this embodiment will be described. FIG. 1(a) is a diagram for explaining the outline and operating environment of the presence area identification system 100 of this embodiment.

[0012] As shown in Fig. 1(a), the presence area identification system 100 of the present embodiment includes an HMD 110, access points 310 arranged in each room 510, and a server 700. The presence area identification system 100 of the present embodiment is a wireless LAN (Local Area Network) system in which the HMD 110 and the server 700 transmit and receive data wirelessly via the access point 310. Hereinafter, in this specification, a space divided by a structure is called a room.

[0013] In the present embodiment, access points (APs) 310 (311, 312) for wireless LAN connection are provided in each room 510 (511, 512). Then, on the server 700 side that manages various data necessary for AR display, the room 510 is identified by the identification information of this access point 310.

[0014] In the present embodiment, when a user wearing an HMD (Head Mounted Display) 110 enters the room 510, the HMD 110 connects to an access point (AP) 310, which is a relay device pre-installed in the room 510, and performs authentication processing. Through this authentication processing, the HMD 110 obtains the identification information of the access point 310. Then, various types of information (display-related information) necessary for AR image display transmitted from the server 700 are managed in association with the identification information of the access point 310. Thereby, in the HMD 110, it is possible to identify which room 510 the information is for.

[0015] Also, on the server 700 side, the display-related information in each room 510 is managed in association with the identification information of the access point 310 installed in each room 510. Based on the identification information of the access point 310 relayed at the time of a data transmission request from the HMD 110, the presence area (room 510) of the HMD 110 is identified, and the display-related information prepared for that room 510 is provided to the source HMD 110.

[0016] Hereinafter, in this embodiment, as display-related information, spatial shape information will be taken as an example for explanation. The spatial shape information is information obtained by mapping data scanned from the arrangement and shape of the walls and installed objects in the room 510. The spatial shape information is used to grasp the internal shape surrounded by the walls in the room 510 and the arrangement and shape of furniture and the like installed in the room 510. With this spatial shape information, AR objects can be placed realistically. Placing with a sense of reality means, for example, not placing AR objects behind the wall.

[0017] Generally, the spatial shape information is generated by scanning the space using various sensors provided in the HMD 110 each time the wearer of the HMD 110 enters a predetermined space such as the room 510, which is a task with a large processing load.

[0018] In this embodiment, the server 700 and the HMD 110 itself utilize the fact that the presence area (room 510) of the HMD 110 can be easily specified. If the spatial shape information of each room 510 has been created in the past, it will be reused.

[0019] Hereinafter, this embodiment will be described by taking the case where there are two rooms (the first room 511 and the second room 512) as an example. The first access point 311 is arranged in the first room 511, and the second access point 312 is arranged in the second room 512. Also, there may not be only one HMD 110. For example, there may be a first HMD 111 and a second HMD 112. Hereinafter, when there is no need to distinguish each room, it will be represented by the room 510. Also, when there is no need to distinguish each access point and each HMD, they will be represented by the access point 310 and the HMD 110, respectively. Each access point 310 can transmit and receive data to and from the server 700 via the network 610.

[0020] [HMD] First, the HMD 110 of this embodiment will be described.

[0021] [Appearance of HMD] Figure 2(a) is an external view of the HMD 110 of this embodiment. The HMD 110 shown in this figure is a see-through type HMD 110. The HMD 110 has a transmissive or semi-transmissive display 173. A wearer wearing the HMD 110 can view virtual objects and images displayed on the display 173 superimposed on the external scene.

[0022] Although not shown in the figures, this embodiment is also applicable to an immersive type HMD. A wearer wearing an immersive type HMD does not directly view the surrounding scenery but views the VR world (virtual reality). Further, the immersive type HMD uses the video-through method, that is, uses the data of the external scene image captured by the out-camera 172 (see Figure 2(b)) and views an image in which virtual objects are superimposed on the external scene image. This embodiment is applicable to any of the above methods, but the see-through type HMD 110 will be described as an example.

[0023] [Example of the hardware configuration of the HMD] Figure 2(b) is a hardware configuration diagram showing an example of the internal configuration of the HMD 110.

[0024] The HMD 110 includes a system bus 113, a main processor 120 connected to the system bus 113, a RAM 141, a ROM 142, a flash memory 143, a GPS (Global Positioning System) receiver 151, a geomagnetic sensor 152, a distance sensor 153, an acceleration sensor 154, a gyro sensor 155, a timer 156, a wireless communication I / F 161, a telephone network communication I / F 162, an in-camera 171, an out-camera 172, a display 173, a microphone 181, a speaker 182, an audio decoder 183, a button switch 191, and a touch panel 192.

[0025] The main processor 120 controls the entire HMD 110 according to a predetermined operation program. The components are not limited to a CPU, an MPU, a dedicated logic circuit, etc.

[0026] The system bus 113 is a data communication path that interconnects the main processor 120 and each component within the HMD 110. The main processor 120 and each component within the HMD 110 transmit and receive various commands, data, etc. via the system bus 113.

[0027] The RAM 141 constitutes a rewritable program working area such as a work area used when the main processor 120 executes various programs.

[0028] The ROM 142 and the flash memory 143 store various programs for realizing the functions of the HMD 110, operation setting values, sensor information including detection values from sensors described later, and various display data such as virtual objects and contents. The ROM 142 and the flash memory 143 are so-called non-volatile storage that retains the stored information even when no external power is supplied to the HMD 110.

[0029] The flash memory 143 stores operation programs downloaded from the network and various data created by the operation programs. Each operation program stored in the flash memory 143 can be updated and functionally expanded by a download process from each server device on the network.

[0030] Furthermore, the flash memory 143 can store contents such as videos, still images, and audio downloaded from the network. Also, it can store data such as videos and still images captured by the in-camera 171 or the out-camera 172.

[0031] The RAM 141, the ROM 142, and the flash memory 143 are an example of storage, and other devices such as semiconductor element memories such as SSD (Solid State Drive) and magnetic disk drives such as HDD (Hard Disc Drive) may be used.

[0032] The main processor 120 acquires sensor information from the GPS receiver 151, the geomagnetic sensor 152, the distance sensor 153, the acceleration sensor 154, and the gyro sensor 155, and also acquires the viewing time measured by the timer 156. Then, it detects the position, inclination, azimuth, movement, etc. of the HMD 110 using the sensor information and the viewing time. Further, the HMD 110 may further include other sensors such as an illuminance sensor, a proximity sensor, and an altitude sensor.

[0033] The wireless communication interface (I / F) 161 is connected to a network 610 such as the Internet via the access point 310, and transmits and receives data to and from each server 700 on the network 610. In this embodiment, the wireless LAN connection with the access point 310 or the like uses millimeter waves (60 GHz band). This is because millimeter waves have strong directivity and the radio waves are easily absorbed by the walls of the room. By using millimeter waves, the HMD 110 can surely establish a wireless LAN connection with the access point 310 installed in the room 510 where it is located. Note that Wi-Fi (registered trademark) or Bluetooth (registered trademark) may be used for the wireless communication I / F 161.

[0034] The telephone network communication I / F 162 uses a third-generation mobile communication system (hereinafter referred to as "3G") such as the GSM (registered trademark) (Global System for Mobile Communications) system, the W-CDMA (Wideband Code Division Multiple Access) system, the CDMA2000 system, or the UMTS (Universal Mobile Telecommunications System) system, or a communication system called LTE (Long Term Evolution), the fourth generation (4G), or the fifth generation (5G), and is connected to a communication network through a base station using a mobile communication network, and transmits and receives information to and from a server on the communication network.

[0035] The wireless communication I / F 161 and the telephone network communication I / F 162 each include an encoding circuit, a decoding circuit, an antenna, and the like.

[0036] Also, the HMD 110 may be provided with other communication interfaces such as an infrared communication interface.

[0037] The in-camera 171 and the out-camera 172 are cameras that input image data of the surroundings or an object by converting the light input from the lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. When there is no need to particularly distinguish between the in-camera 171 and the out-camera 172, they are collectively referred to as the camera 170.

[0038] The display 173 is a display device such as a liquid crystal panel, and provides image data to the wearer of the HMD 110. The HMD 110 includes a video RAM (not shown). Based on the image data input to the video RAM, virtual objects and videos are displayed on the screen of the display 173.

[0039] The microphone 181 converts the voice of the wearer of the HMD 110 or the like into audio data and inputs it.

[0040] The speaker 182 outputs audio information and the like.

[0041] The audio decoder 183 performs decoding processing of the encoded audio signal as necessary.

[0042] The button switch 191 and the touch panel 192 are operation devices for inputting operation instructions to the HMD 110. The operation device is not limited to the button switch 191 and the touch panel 192. For example, an operation signal of the HMD 110 may be transmitted from a separate portable terminal device (such as a smartphone or a tablet terminal) connected by wired communication or wireless communication, the HMD 110 may receive the operation signal, and operate according to this operation signal. Also, voice may be input from the microphone 181, the main processor 120 may execute voice recognition processing to generate an operation signal, and the operation control of the HMD 110 may be performed.

[0043] Note that the configuration example of the HMD 110 shown in Fig. 2(a) includes configurations that are not essential for the present embodiment, but even if these configurations are not provided, the effects of the present embodiment will not be impaired. Further, configurations not shown, such as a digital broadcast reception function and an electronic money settlement function, may be further added.

[0044] [Function Blocks of HMD] Next, the functional configuration of the HMD 110 of the present embodiment will be described. Fig. 3(a) is a functional block diagram of the HMD 110 of the present embodiment.

[0045] The HMD 110 of the present embodiment includes each function of a main control unit 211, an authentication unit 212, a data management unit 213, a spatial shape information acquisition unit 214, an AR image processing unit 215, and a communication unit 216, and an HMD-side spatial shape information table (HMD-side library) 221 and an authentication information table 222. Each function is realized by the main processor 120 loading a program stored in the ROM 142 or the flash memory 143 into the RAM 141 and executing it. Further, the HMD-side library 221 is constructed on the ROM 142 or the flash memory 143.

[0046] The main control unit 211 controls the operations of each part of the HMD 110.

[0047] The authentication unit 212 accesses the access point 310 arranged in each room 510 and performs authentication. In the present embodiment, when receiving the network name (wireless LAN network name) of the access point 310 and the information for specifying the access point 310 (access point ID) via the wireless communication I / F 161, it transmits the password required for network connection to the access point 310. The password is stored in the authentication information table 222 in advance. When receiving a notification of successful authentication from the access point 310, it notifies other functions such as the main control unit 211.

[0048] In the authentication information table 222, passwords, device names, etc. necessary for various types of authentication are registered. In the present embodiment, for example, an authentication password required when connecting to the access point 310 is stored.

[0049] The data management unit 213 manages the spatial shape information and the like of the HMD-side library 221. The spatial shape information is acquired by the spatial shape information acquisition unit 214 described later.

[0050] In the present embodiment, the data management unit 213 first determines whether there is spatial shape information for the room 510 (hereinafter also referred to as the own room) where the HMD 110 currently exists. The spatial shape information of each room 510 is stored in the HMD-side library 221 in association with the access point ID arranged in the room.

[0051] Here, FIG. 3(b) shows an example of the HMD-side library 221 of the present embodiment. As shown in this figure, in the HMD-side library 221, for each room 510, spatial shape information 221b is registered in association with the access point ID 221a. In the HMD-side library 221, display-related information such as AR image information 221c may also be registered.

[0052] In the present embodiment, as the access point ID 221a, information for specifying the access point 310 acquired at the time of authentication is used. As the access point ID 221a, for example, the MAC address of the access point 310 may be used.

[0053] Note that, for example, when spatial shape information is not stored in association with the access point ID 221a for the room 510, the data management unit 213 requests the server 700 for the spatial shape information of the room 510. The request is made via the access point 310.

[0054] When the data management unit 213 receives the spatial shape information from the server 700 via the access point 310, it stores the received spatial shape information 221b in the HMD-side library 221 in association with the access point ID 221a. On the other hand, when receiving a reply from the server 700 indicating no registration of the spatial shape information, it instructs the spatial shape information acquisition unit 214 to acquire the spatial shape information.

[0055] The data management unit 213 obtains the spatial shape information of the room 510 stored in the server 700 and stores it in the HMD-side library 221.

[0056] When the spatial shape information acquisition unit 214 receives an instruction to acquire the spatial shape information from the data management unit 213, it creates the spatial shape information of the area (room 510) where the self-device (HMD110) currently exists. The created spatial shape information 221b is registered in the HMD-side library 221. At this time, the spatial shape information acquisition unit 214 registers it in association with the access point ID 221a acquired by the authentication unit 212.

[0057] In the present embodiment, the spatial shape information acquisition unit 214 uses the camera 170 to photograph while scanning the walls of the room 510, the arrangement and shape of the installed objects, etc. Then, the obtained photographed data is mapped to acquire the spatial shape information of the room 510.

[0058] The AR image processing unit 215 generates an AR image using the AR image information 221c and displays it on the display 173. The display is performed when the display position of the generated AR image exists within the field of view of the wearer of the HMD110. The HMD110 itself determines whether the display position of the AR image exists within the field of view of the HMD110.

[0059] The AR image information 221c may be acquired from the server 700, for example, and stored in the HMD-side library 221. Also, the field of view information of the HMD110 may be transmitted to the server 700, and the server 700 may determine the display position of the AR image and transmit the necessary AR image information 221c to the HMD110.

[0060] The communication unit 216 transmits and receives data to and from an external device via the wireless communication I / F 161. In this embodiment, data is transmitted toward the access point 310 via the wireless communication I / F 161. Also, data transmitted from the access point 310 is received.

[0061] Note that the HMD 110 of this embodiment may also include, for example, a photographed data storage unit that stores photographed data acquired by the camera 170. Also, all data other than the authentication information may be configured to be stored externally. For example, it may be a device such as a server connected via a network. In this case, it is acquired via the wireless communication I / F 161 as necessary.

[0062] [Access Point] The access point 310 has the same configuration as a conventionally used access point. In this embodiment, it is a wireless device having a router function for connecting a wireless LAN client such as the HMD 110 to another network.

[0063] In this embodiment, as described above, on the server 700 side, information for identifying the access point 310 is used to identify in which room 510 the HMD 110 is present. For this reason, at least one access point 310 is provided in each room 510. Also, in this embodiment, for example, the wireless LAN connection between the HMD 110 and the access point 310 uses millimeter waves (60 GHz band) so as to be difficult to connect to the access point 310 in other rooms.

[0064] Note that the wavelength of the electromagnetic wave to be used is not limited to this. Any electromagnetic wave in a wavelength band that is difficult to connect to the access point 310 in other rooms and can surely be connected to the access point 310 in its own room may be used.

[0065] Also, it is desirable that the access point 310 be installed near the center of the ceiling of each room 510 so that the HMD 110 can connect to the access point 310 regardless of the position of the wearer of the HMD 110 in the room 510.

[0066] Each access point 310 is set with a MAC (Media Access Control) address which is a unique physical address. Therefore, the MAC address may be used as the access point ID for identifying the access point 310. Instead of the MAC address, an IP (Internet Protocol) address which is a logical address or a unique access point name may also be used. Hereinafter, in this embodiment, the case of using the MAC address will be taken as an example for explanation.

[0067] [Server] Next, the server 700 of this embodiment will be described. The server 700 of this embodiment holds information related to AR display as display-related information for each room 510 to be managed. Then, in response to a request from the HMD 110, it provides the display-related information of the room 510 where the requesting HMD 110 exists. This function is the same as that of a conventional AR server.

[0068] [Hardware Configuration of Server] FIG. 4(a) is an example of the hardware configuration of the server 700 of this embodiment. As shown in this figure, the server 700 includes a processor (main processor) 720, a storage device (server storage device) including a RAM 741, a ROM 742, a flash memory 743, etc., and a communication interface (I / F) 761, like a general server. In addition, a display 773, an operation device 791, etc. may be provided. Since these are the same as the components with the same names in the HMD 110, the description thereof will be omitted here.

[0069] [Functional Configuration of Server] FIG. 4(b) is a functional block diagram of a part related to the processing of the server 700 according to the present embodiment. As shown in this figure, the server 700 includes a spatial shape information management unit 721 and a server-side spatial shape information (hereinafter referred to as a server-side library) 730.

[0070] The spatial shape information management unit 721 manages the data in the server-side library 730. In the present embodiment, in response to a request from the HMD 110, it provides the spatial shape information of the room 510 where the accessed access point 310 passes through. When providing, it checks whether the spatial shape information is registered in the server-side library 730 in association with the access point ID (MAC address) of the access point 310 included in the request data. If it is registered, the spatial shape information is provided. On the other hand, if it is not registered, data indicating that it is not registered is returned.

[0071] Furthermore, in response to the return of data indicating that it is not registered, when the spatial shape information is transmitted, the spatial shape information is registered in the server-side library 730 in association with the access point ID of the relayed access point 310.

[0072] Here, an example of the server-side library 730 is shown in FIG. 4(c). As shown in this figure, display-related information (including spatial shape information) of each room 510 is registered in the server-side library 730. The server-side library 730 includes, for each room 510, a room number 731 for identifying the room, an access point ID 732, the spatial shape information 733 of the room 510, and information on a computer-generated augmented reality image (AR image) (AR image information) 735. In addition, it may include voice information related to the room 510 and other information related to the room 510. Also, the room number 731 may not be provided.

[0073] In the server - side library 730 shown in this figure, for the data where the room number 731 is "Room 511", the access point ID 732 is "Adrs01", the spatial shape information 733 is "Map01", the AR image information 735 is "AR01", the audio information is "Snd01", and other information is "Othr01". Similarly, for rooms from Room 512 to Room n, the data shown in this figure is stored.

[0074] In this embodiment, the server 700 is provided with such a server - side library 730. Thus, if the access point ID of the access point 310 is identified, Room 510 can be specified, and all information regarding that Room 510 can be obtained.

[0075] Note that in the server - side library 730, basically, the spatial shape information 733, AR image information 735, audio information, and other information regarding each room 510 are preferably acquired in advance and registered in association with the room number 731 and the access point ID 732. In particular, for the spatial shape information 733, it is desirable to create and register the spatial shape information of each room 510 in advance using a standard HMD 110 or the like.

[0076] However, in this embodiment, when the data of the access point ID 732 of the relay access point 310 itself is not registered, the transmitted spatial shape information is newly registered. That is, in this embodiment, the server - side library 730 can be supplemented later.

[0077] Note that the server - side library 730 is stored in the storage device of the server 700.

[0078] [Spatial Shape Information Acquisition Process] Hereinafter, the flow of the initial process (spatial shape information acquisition process) when the HMD 110 enters Room 510 by the presence area identification system 100, which is an information processing system capable of identifying the HMD presence area in this embodiment, will be described.

[0079] As described above, usually, when the wearer of the HMD 110 enters a new room 510, the HMD 110 uses various sensors it is equipped with to first collect the spatial shape information of that room 510 each time. However, in this embodiment, since the existence area of the HMD can be specified, the spatial shape information created in the past can be managed in association with the room 510 (access point 310 arranged therein), and it can be used later.

[0080] Hereinafter, the spatial shape information acquisition process of this embodiment will be described with reference to FIG. 5.

[0081] This process starts when the HMD 110 is activated. Alternatively, it is started by an instruction from the wearer. For example, when the wearer enters a predetermined room 510, the wearer gives an instruction to the HMD 110 to perform the initial spatial shape information acquisition process. Alternatively, the HMD 110 may be configured to detect the network name of the access point 310 and automatically start this process.

[0082] First, the authentication unit 212 performs an authentication process for connecting to the wireless LAN access point 310 (step S1101). Details of the authentication process will be described later.

[0083] After the connection to the wireless LAN via the access point 310 is confirmed through the authentication process by the authentication unit 212, the data management unit 213 determines whether the spatial shape information of this room 510 is held in the HMD 110 (step S1102).

[0084] If it is held (S1102; Yes), the process ends as it is.

[0085] On the other hand, if it is not held (S1102; No), the data management unit 213 requests the spatial shape information of this room 510 from the server 700 via the access point 310 (step S1103). Then, the data management unit 213 determines the content of the reply in response to the request (step S1104).

[0086] If there is spatial shape information in the reply (S1104; Yes), the data management unit 213 acquires (downloads) the provided spatial shape information from the server 700 (step S1105), and registers the acquired spatial shape information in the HMD-side library 221 (step S1108). At this time, the data management unit 213 registers the spatial shape information 221b in association with the access point ID 221a of the access point 310 acquired by the authentication unit 212, and ends the process.

[0087] On the other hand, if the reply does not include spatial shape information (S1104; No), the data management unit 213 instructs the spatial shape information acquisition unit 214 to acquire spatial shape information.

[0088] In response to the instruction, the spatial shape information acquisition unit 214 scans the room 510 (step S1106) and acquires the mapped spatial shape information. Then, the acquired spatial shape information is transmitted to the server 700 (step S1107), and is registered in the HMD-side library 221 in association with the access point ID of the access point 310 acquired by the authentication unit 212 (step S1108). When transmitting to the server 700, it is transmitted via the access point 310. Then, the process ends.

[0089] Next, the data transmission and reception between the devices during the authentication process will be described. FIG. 6 is a time chart for explaining the data transmission and reception between the first HMD 111 and the first access point 311 in the first room 511 of FIG. 1 and the server 700.

[0090] It is composed of a total of three time axes: the time axis T451 processed by the first HMD 111, the time axis T452 processed by the first access point 311, and the time axis T453 processed by the server 700.

[0091] From each access point 310, the network name of the access point 310 (the network name of the wireless LAN) is constantly transmitted to the terminals capable of connecting to the wireless LAN including the HMD 110. Therefore, the network name of the access point is also transmitted from the first access point 311 (step S1301).

[0092] The first HMD 111 detects the transmitted network name (step S1201), and selects the network name targeted by the wearer of the first HMD 111 from the obtained network name group as the connection target (step S1202). In the present embodiment, only one access point 310 is installed in one room 510. Also, communication between the HMD 110 and the access point 310 uses millimeter waves that make it difficult to communicate across the room 510. Therefore, in the present embodiment, the first HMD 111 can detect only the network name transmitted by the first access point 311 installed in the first room 511 and select that network name as the connection target.

[0093] Subsequently, the first HMD 111 makes an authentication request to the first access point 311. Here, the password required for network connection is notified to the first access point 311 (step S1203). The password is acquired by the authentication unit 212 from the authentication information table 222 and transmitted.

[0094] The first access point 311 performs password authentication using the received password (step S1302). The first access point 311 verifies whether the pre-registered password required for connection to each connectable network matches the received password. If they match, the authentication is successful. When the authentication is successful, the first HMD 111 and the first access point 311 are connected via the wireless LAN. The first access point 311 transmits a signal (authentication OK) permitting authentication to the first HMD 111 (step S1303).

[0095] Since the authentication process between the first HMD 111 and the first access point 311 is the same as that of a general wireless LAN connection, detailed description thereof is omitted here.

[0096] In order to strengthen the security of the network, not only password authentication but also ID (Identity Document) authentication may be added. Alternatively, an authentication server that performs both ID and password authentication may be added, and password authentication may be performed by this authentication server.

[0097] After the connection is established between the first HMD 111 and the AP 311, that is, when the authentication is successful in step S1302, the first access point 311 transmits identification information for identifying the first access point 311 to the server 700 (step S1304). In this embodiment, a MAC address, which is a device-specific physical address, is used as this identification information. If the password authentication in step S1302 fails, the process in step S1304 is not performed.

[0098] In the server 700, the space shape information management unit 721 verifies whether the received MAC address is registered in the server-side library 730.

[0099] If the MAC address is not registered, the space shape information management unit 721 determines that it is a new MAC address (new room), and registers the MAC address in the server-side library 730 as an access point ID 732 (step S1401). As a result, the MAC address of the first access point 311 disposed in the first room 511 where the first HMD 111 is present is registered in the server 700. In addition, the server 700 recognizes that the first HMD 111 is present in the first room 511 where the first access point 311 having the newly registered MAC address is disposed.

[0100] In step S1401, if it is determined that the MAC address is registered, the server 700 recognizes that the first HMD 111 exists in the first room 511 where the first access point 311 having the corresponding MAC address is arranged.

[0101] Thereafter, when a request for providing spatial shape information is received from the first HMD 111 via the first access point 311, the spatial shape information management unit 721 checks whether there is corresponding spatial shape information. The check is performed by determining whether the spatial shape information 733 is registered in the server-side library 730 in association with the MAC address of the source first access point 311.

[0102] If the spatial shape information management unit 721 determines that the information is registered, it transmits the spatial shape information 733 to the source. On the other hand, if it is not registered, it transmits a message indicating that it is not registered to the source.

[0103] In addition, after the spatial shape information management unit 721 transmits a message indicating that the information is not registered, if it receives spatial shape information via the first access point 311, it registers the received spatial shape information 733 in the server-side library 730 in association with the MAC address of the first access point 311.

[0104] As described above, according to this embodiment, when the wearer of the HMD 110 enters the room 510, the HMD 110 receives connection authentication from the access point 310 in the room, and on the HMD 110 side, associates the room with the access point. Further, when connection authentication is performed on the HMD 110, the access point 310 transmits the MAC address as the identification information of the access point 310 to the server 700. On the server side, by receiving the MAC address, it is recognized that the HMD 110 exists in the room 510 registered in association with the MAC address. The MAC address and the room 510 are held by the server 700 in advance as the server-side library 730 in association with each other.

[0105] In this way, in the present embodiment, on the server 700 side, the presence area of the HMD 110, that is, the room 510 where the HMD 110 is present, is specified using the identification information of the access point used in the wireless LAN. As a result, it is possible to easily grasp the presence area of the HMD 110 only with existing devices. That is, the presence area of the HMD 110 can be specified without adding special hardware.

[0106] Further, according to the present embodiment, the HMD 110 can provide the information necessary to specify the presence area of the HMD 110 to the server 700 side only by performing the normal authentication process with the access point 310. That is, the HMD 110 can cause the server 700 to grasp the room 510 where the own device is present without giving a new instruction or the like to the wearer.

[0107] Furthermore, according to the present embodiment, the area (room 510) where the HMD 110 is present is specified by the information uniquely identifying the access point 310 arranged in each room 510. Therefore, there is no restriction in the height direction as in the case of position specification by GPS. That is, in the case of GPS, in the case of a building with multiple floors or the like, even if the floors are different, if the positions on the plane are the same, it is difficult to distinguish. However, according to the present embodiment, even in such a situation, the area where the HMD 110 is present can be accurately specified.

[0108] In this way, according to the present embodiment, the server 700 can easily specify the presence area of the HMD 110 with sufficient accuracy. Therefore, by simply managing various display-related information required for AR display created in the past for each presence area, it can be appropriately provided to the HMD 110. Also, the HMD 110 can appropriately receive the display-related information managed by the server 700.

[0109] In particular, the spatial shape information needs to be generated each time the wearer of the HMD 110 moves through the room 510, which places a heavy burden on the processing of the HMD 110. However, according to the present embodiment, since the server 700 can easily identify the room 510 into which the HMD 110 has entered, if the server 700 manages the spatial shape information created in the past, it can be provided from the server 700. Therefore, according to the present embodiment, the processing load when the HMD 110 enters the room can be significantly reduced.

[0110] <Modification Example 1> In the above embodiment, the access point 310 using the wireless LAN is used to identify the room 510 in which the HMD 110 is present. However, the relay device used to identify the room 510 is not limited to the access point 310. For example, a mobile phone base station may be used.

[0111] Hereinafter, a method for identifying the room 510 in which the HMD 110 is present using a mobile phone base station as a relay device will be described. FIG. 7(a) is a diagram for explaining the outline of this modification example.

[0112] The presence area identification system 100a of this modification example basically has the same configuration as the above embodiment, except that a mobile phone base station (hereinafter referred to as BS) 320 is used instead of the access point 310 used as the relay device in the above embodiment. In this figure, a case where the first BS 321 is arranged in the first room 511 and the second BS 322 is arranged in the second room 512 is illustrated.

[0113] Therefore, the network to which the BS 320 is connected is not the network 610 such as the Internet, but the mobile phone line network 620. And in order to connect to the server 700 via this mobile phone line network 620, a mobile management 631 is provided between the two.

[0114] Mobile Management 631 is located at the core of the mobile phone line network and manages the mobile phone line network together with the Home Subscriber Server 632 that stores subscriber information. The BS 320 provided in each room 510 is connected to Mobile Management 631 via the mobile phone line network 621. Then, each BS 320 is connected to the network 610 via Mobile Management 631 and accesses the server 700.

[0115] Note that Mobile Management 631 is connected to a plurality of mobile phone base stations (not shown) and is also connected to another mobile phone line network 622.

[0116] In this modification, the wireless connection between the BS 320 and the HMD 110 as a mobile phone is performed using millimeter waves (28 GHz band), which are the 5G standard for mobile phones. Millimeter waves have strong directivity and are easily absorbed by the walls of the room 510. The HMD 110 present in the room 510 is surely wirelessly connected to the BS 320 installed in the room 510 and will not be wirelessly connected to the BS 320 installed in another room 510 across the wall.

[0117] A unique mobile phone base station identification number (BS-ID) is set for each BS 320. Therefore, if the correspondence between the room 510 and the BS-ID of the BS 320 installed in that room 510 is established, the room 510 can be specified by the BS-ID of the BS 320.

[0118] Hereinafter, this modification will be described with a focus on the points different from the first embodiment.

[0119] The HMD 110 is basically the same as in the first embodiment. However, as shown in Fig. 8(a), the spatial shape information (HMD-side library 221) of each room 510 held by the HMD 110 is stored in association with the BS-ID 221d of the BS 320 arranged in each room 510.

[0120] Similarly, the configuration of the server 700 is basically the same. However, as shown in FIG. 8(b), the server-side library 730 is also managed in association with the BS-ID 732b instead of the MAC address of the access point 310.

[0121] Note that the access point 310 or the BS 320 may be arranged in the room 510, and there are different cases. In this case, for each room 510, the spatial shape information is stored in association with the identification information of the relay device arranged in the room 510.

[0122] The spatial shape information acquisition process in the HMD 110 is basically the same as that in the above-described embodiment. However, in the authentication process, instead of the authentication process for connecting to the access point 310, a process for connecting to the BS 320 is performed.

[0123] Hereinafter, the data transmission and reception between the devices in the authentication process of this modification example will be described.

[0124] FIG. 9 is a timing chart of the authentication process of this modification example. Here, the data transmission and reception between the present embodiment, the first BS 321, the home subscriber server 632, and the server 700 will be described. This figure is composed of a total of four time axes, namely, the time axis T421 processed by the first HMD 111, the time axis T422 processed by the BS 311, the time axis T423 processed by the home subscriber server 632, and the time axis T424 processed by the server 700.

[0125] Generally, each BS 320 always notifies the carrier identification number and the mobile phone base station identification number (BS-ID) to the mobile phone terminal (HMD 110) that can be connected to the BS 320. Also in this modification example, the carrier identification number and the BS-ID are notified from the first BS 321 to the first HMD 111 (S1601).

[0126] In the first HMD 111, the notified business operator identification number is detected, and the target business operator is selected (step S1501). Since the corresponding mobile phone operator information is stored in the SIM card built into the first HMD 111, the BS 311 provided by the corresponding business operator is selected using this information. The first HMD 111 stores the business operator identification number and the BS-ID of the selected BS 311.

[0127] In the first HMD 111, the identification number of the first HMD 111 when used as a mobile phone is stored in the SIM card within the device itself. The first HMD 111 transmits the identification number to the home subscriber server 632 via the BS 311 (step S1502).

[0128] The home subscriber server 632 performs user authentication using the identification number of the first HMD 111 received in step S1502 (step S1701). User authentication is performed by comparing the identification number with the subscriber information pre-registered in the home subscriber server 632 to check if they match. If the transmitted identification number and the subscriber information match, authentication is successful, and the home subscriber server 632 transmits a signal (authentication OK) indicating successful authentication to the first HMD 111 (step S1702).

[0129] Since the authentication process between the first HMD 111 and the first BS 321 described above is a general mobile phone connection process, detailed description is omitted here.

[0130] Furthermore, in step S1701, if authentication is successful, the first HMD 111 transmits a spatial shape information request to the server 700 together with the received BS-ID (step S1503).

[0131] In response to this request, the server 700 may request a password from the source first HMD 111 in order to confirm whether to send the spatial shape information to the source first HMD 111 (step S1801). In response, the first HMD 111 sends a password to the server 700 (step S1504). Note that the password for verification is registered in the server 700 in advance.

[0132] The server 700 performs authentication with the sent password (step S1802), and if the password matches, it sends a signal (authentication OK) indicating successful authentication to the first HMD 111 (step S1803).

[0133] If the authentication at the server 700 is successful, hereafter, similar to the above embodiment, the first HMD 111 acquires the spatial shape information of the first room 511 where it is located from the server 700 as necessary.

[0134] Thus, according to this modification example, by using the base station of the mobile phone network used in the mobile phone, the server 700 specifies the existence area (room 510) of the HMD. Thereby, similar to the above embodiment, the existence area of the HMD 110 can be easily grasped only by existing devices. That is, the existence area of the HMD 110 can be specified without adding special hardware.

[0135] Also, according to this modification example, similar to the above embodiment, the HMD 110 only performs the authentication normally performed for communication, and the server 700 can grasp the existence area of the HMD 110. That is, the HMD 110 can provide the server 700 with the information necessary to specify the existence area of its own device without giving a new instruction or the like to its wearer.

[0136] Thereby, according to this modification example, the same effects as the above embodiment can be obtained.

[0137] <Modification Example 2> Note that in this modification example, it is premised on the use of millimeter waves with strong directivity and a high attenuation rate, but it is not limited to this.

[0138] In recent years, a mobile phone network configured with femtocells using low-power mobile phone base stations has been established. Since the communication range in a femtocell is extremely narrow compared to a conventional cell, the output from the devices within room 510 is very likely to be received by the mobile phone base station within room 510. Therefore, a mobile phone base station using such a femtocell may be used.

[0139] <Modification Example 3> Note that in the above embodiment, the access point 310 performs a wireless LAN connection and accesses the server 700 via a network 610 such as the Internet network, but it is not limited thereto. For example, as shown in FIG. 1(b), instead of the network 610, a wired LAN network 630 may be used. In this case, instead of the access point 310, a relay device such as a hub (HUB) 340 (341, 342) is used.

[0140] By using a wired LAN, in addition to the effects of the above embodiment and the modification example, an effect of reducing the risk of information leakage due to the Internet network can be obtained.

[0141] <<Second Embodiment>> Next, a second embodiment of the present invention will be described. In this embodiment, a plurality of access points 330 are provided within the same room 510. And among the access points 330, there are access points with different performances.

[0142] Hereinafter, this embodiment will be described focusing on the configuration different from the first embodiment.

[0143] FIG. 10(a) is a diagram for explaining the outline of the operating environment of the presence area specifying system 100 of this embodiment. Here, for the sake of simplicity of explanation, a case where two systems, an access point 331 and an access point 332, are provided as a plurality of access points 330 will be described as an example. It may be configured to be connectable by wireless LANs with three or more different transmission standards.

[0144] Figure 10(b) is a table 520 showing the performance differences between each access point 331 and 332. There are differences in performance between the two, such as connection speed and the presence or absence of data volume limits. This information may be stored, for example, in the HMD 110 or may be held by the server 700. Note that a difference in the charging amount may be set between the two.

[0145] As shown in this figure, the access point 331 has a medium connection speed, a limit on the data volume, and also has advertisements. On the other hand, the access point 332 enables high-speed connection, has no data volume limit, and has no advertisements.

[0146] Therefore, for example, when transmitting and receiving spatial shape information or AR image information, it is possible to transmit and receive more detailed data in a shorter time via the access point 332.

[0147] Also, in this embodiment, when the HMD 110 is connected via any access point 330 in the room 510, the server 700 determines that the HMD 110 is present in the room 510. However, as the server-side library 730, spatial shape information and AR image information with different resolutions may be held for each access point 330 with different performance.

[0148] An example of the server-side library 730 in this case is shown in Figure 11. As shown in this figure, information is held for each access point 330 of each room 510. Specifically, it includes the access point ID of each access point 330, spatial shape information 733, information indicating its resolution (hereinafter simply referred to as resolution) 734, AR image information 735, and its resolution 736. Here, although the case of changing the quality (resolution) of the AR image information provided according to the performance of the access point 330 is taken as an example for explanation, the quality of the AR image information may be of one type.

[0149] The space shape information management unit 721 of the server 700 transmits the space shape information held in the server-side library 730 according to the relayed access point 330.

[0150] As a result, when the HMD 110 requests space shape information via a more performant access point 330, it can obtain higher-resolution and high-quality space shape information.

[0151] Note that the wearer of the HMD 110 may be able to select which access point 330 to request space shape information from the server 700 via, or it may be predetermined within the HMD 110.

[0152] When the wearer makes a selection, if the HMD 110 detects a plurality of access points 330, it displays a list thereof on the display 173 and accepts the selection.

[0153] When it is predetermined, selection rules are determined in advance and registered in the ROM 142, flash memory 143, etc. The selection rules are, for example, prioritize the most performant among free ones, prioritize the most performant regardless of free or paid, etc. Accordingly, the authentication unit 212 selects the access point 330 to be the relay device.

[0154] According to the present embodiment, similar to the first embodiment, the server 700 grasps the presence area of the HMD 110 using the access point IDs of the access points 330 arranged in each room 510. Therefore, the same effects as in the first embodiment can be obtained.

[0155] Furthermore, according to the present embodiment, when there are access points 330 with different performances in one room 510, the server 700 manages them separately. And it provides display-related information of quality according to the performance of the access point 330 that the HMD 110 has performed connection authentication for.

[0156] As described above, according to this embodiment, the server 700 can easily identify the existence area of the HMD 110 with necessary and sufficient accuracy, and can also grasp the data quality desired by the HMD 110. Therefore, by simply managing various display-related information required for AR display, which was created in the past, for each existence area and for each quality, it can be appropriately provided to the HMD 110. Also, the HMD 110 can appropriately receive the display-related information managed by the server 700.

[0157] <<Third Embodiment>> Next, a third embodiment of the present invention will be described. In this embodiment, the spatial shape information held by the server 700 includes its resolution. Then, according to the performance of the HMD 110, the spatial shape information of the server 700 is overwritten.

[0158] Hereinafter, this embodiment will be described with a focus on the configuration different from the first embodiment.

[0159] The HMD 110 of this embodiment has basically the same configuration as that of the first embodiment. However, when the spatial shape information acquisition unit 214 of this embodiment acquires spatial shape information, it also specifies its resolution.

[0160] Also, as shown in FIG. 12(a), the HMD-side library 221 of this embodiment also registers the resolution 221e of the spatial shape information 221b in association with the access point ID 221a.

[0161] The data management unit 213 of this embodiment performs basically the same processing as that of the first embodiment. Further, the data management unit 213 of this embodiment grasps in advance the best resolution of the spatial shape information that can be acquired by the spatial shape information acquisition unit 214. Note that the performance of the HMD 110 is determined by, for example, the spatial resolution of the sensor. Then, by comparing with the resolution of the spatial shape information of the self-room held in the server-side library, it is determined whether to acquire from the server 700 or to acquire with the self-device.

[0162] That is, when the data management unit 213 of the present embodiment acquires the spatial shape information from the server 700, it compares the resolution thereof with the resolution (optimal resolution) of the spatial shape information that can be generated by the own device. And if the optimal resolution is higher, the spatial shape information acquisition unit 214 is made to generate the spatial shape information by the own device. Then, together with the information indicating the resolution, it is transmitted to the server 700.

[0163] Also, the server 700 of the present embodiment basically has the same configuration as that of the first embodiment. However, the server-side library 730 has the resolution of the spatial shape information.

[0164] FIG. 12(b) is an example of the server-side library 730 of the present embodiment. As shown in this figure, in addition to the information held by the server-side library 730 of the first embodiment, it includes the resolution 734 of the spatial shape information and the resolution 736 of the AR image information.

[0165] Note that information regarding the optimal resolution of the spatial shape information that can be acquired by the spatial shape information acquisition unit 214 of the own device is held in advance.

[0166] When the spatial shape information management unit 721 of the server 700 receives the spatial shape information, it determines whether to register it in the server-side library 730 in association with the access point ID of the access point 310 through which the transmission source passed.

[0167] In the present embodiment, if it is not registered, the received spatial shape information is registered. On the other hand, when the spatial shape information is already registered in association with the access point ID, its resolution is compared with the resolution of the received spatial shape information. And if the resolution of the received spatial shape information is higher, the existing spatial shape information is replaced with the received spatial shape information. The resolution is also replaced with the resolution of the received spatial shape information.

[0168] Thereby, the server-side library 730 of the present embodiment holds the spatial shape information having the best resolution at that time.

[0169] The processing flow of the spatial shape information acquisition process of the present embodiment will be described below. FIG. 13 shows the processing flow of the spatial shape information acquisition process by the HMD 110 of the present embodiment. This process is started by the same trigger as in the first embodiment. First, the authentication unit 212 performs an authentication process (step S1101). This process is the same as that in the first embodiment.

[0170] The data management unit 213 determines whether the spatial shape information of this room 510 is held in the HMD 110, in the same manner as in the first embodiment (step S1102).

[0171] If it is held (S1102; Yes), the process ends as it is.

[0172] On the other hand, if it is not held (S1102; No), the data management unit 213 requests the server 700 for spatial shape information via the access point 310 (step S1103). Then, the data management unit 213 determines the content of the reply in response to the request (step S1104).

[0173] If the reply contains spatial shape information (S1104; Yes), the data management unit 213 downloads the provided spatial shape information from the server 700 (step S1105).

[0174] Then, the data management unit 213 of the present embodiment compares it with the best resolution of the spatial shape information that can be acquired by the spatial shape information acquisition unit 214 of its own device (step S4101). If the resolution of the spatial shape information of its own device is equal to or lower than the resolution of the spatial shape information held by the server 700 (S4101; No), as in the first embodiment, the received spatial shape information is registered in the HMD - side library 221 (step S1108), and the process ends. At this time, the data management unit 213 registers the spatial shape information in association with the access point ID of the access point 310 acquired by the authentication unit 212.

[0175] On the other hand, when the reply does not include spatial shape information (S1104; No), and when the best resolution of the spatial shape information of the own device is higher than the spatial shape information held by the server 700 (S4101; Yes), the data management unit 213 instructs the spatial shape information acquisition unit 214 to acquire the spatial shape information.

[0176] Upon receiving the instruction, the spatial shape information acquisition unit 214 scans the room 510 (step S1106) to acquire the spatial shape information. Then, the acquired spatial shape information is transmitted toward the server 700 (step S1107), and the acquired spatial shape information is registered in the HMD side library 221 in association with the access point ID of the access point 310 acquired by the authentication unit 212 (step S1108), and the process ends. When transmitting to the server 700, it is transmitted via the access point 310.

[0177] Next, the spatial shape information management process on the server side of the present embodiment will be described. FIG. 14 is a processing flow of the spatial shape information management process by the spatial shape information management unit 721 of the server 700 of the present embodiment. This process starts when the spatial shape information is received from the HMD 110.

[0178] The spatial shape information management unit 721 determines whether the spatial shape information is already registered in the server side library 730 for the room 510 specified by the received spatial shape information (step S4201). Here, it is determined whether the spatial shape information is already registered in association with the access point ID of the relayed access point 310.

[0179] If it is not registered (S4201; No), the received spatial shape information is registered in association with the access point ID of the relayed access point 310 (step S4203). At this time, the resolution is also registered together.

[0180] On the one hand, if it has already been registered (S4201; Yes), the resolution of the corresponding spatial shape information in the library is compared with the resolution of the received spatial shape information (step S4202).

[0181] If the resolution of the received spatial shape information is higher than the resolution of the spatial shape information in the library (S4202; Yes), the spatial shape information in the library is replaced with the received spatial shape information (step S4203), and the process ends. At this time, the resolution is also set to the resolution of the received spatial shape information.

[0182] On the other hand, in other cases (S4202; No), the transmitted spatial shape information is discarded and the process ends.

[0183] As described above, in this embodiment, the display-related information (spatial shape information) in the server-side library 730 is held together with its resolution. And when it is possible to obtain higher-resolution information on the HMD110 side, it is replaced with the display-related information (spatial shape information) obtained on the HMD110 side.

[0184] Therefore, according to this embodiment, on the server 700 side, high-resolution display-related information (spatial shape information) obtained by the HMD110 with the highest performance among the HMD110s that may be used is stored. And the HMD110 can use the high-resolution spatial shape information.

[0185] Therefore, according to this embodiment, the HMD110 can use high-resolution spatial shape information regardless of the spatial resolution of the sensors of its own device.

[0186] <Modification Example 4> In the above embodiment, when the resolution of the received spatial shape information is high on the server 700 side, the already registered spatial shape information is replaced, but it is not limited to this. For example, on the server side, spatial shape information of multiple resolutions may be held.

[0187] In this case, when transmitting to the HMD 110, the HMD 110 side may be configured to accept a resolution specification.

[0188] <<Fourth Embodiment>> Next, a fourth embodiment of the present invention will be described. In the above-described embodiments and their modifications, the server 700 side discriminates the presence area of the HMD 110 by necessarily connecting to the relay device installed in each room 510 and using the identification information of the relay device. However, in a general communication environment, the HMD 110 is not necessarily connected to the relay device arranged in the room 510 where its wearer is present.

[0189] In the present embodiment, even in such an environment, a method capable of specifying the presence area of the HMD 110 is provided on the server 700 side. In the present embodiment, a server-side library 730 that associates a room 510, an access point ID of an access point arranged in the room 510, and spatial shape information of the room 510 is created in advance and held by the server 700.

[0190] Hereinafter, the present embodiment will be described focusing on points different from the first embodiment.

[0191] FIG. 15 is an explanatory diagram for explaining the outline of the present embodiment.

[0192] Normally, the electric field strength of the wireless LAN generated from the first access point 311 in the same first room 511 is stronger than the electric field strength of the wireless LAN generated from the second access point 312 in the other second room 512. Therefore, the first access point 311 in the same first room 511 is automatically selected as the access point to which the HMD 110 is connected.

[0193] However, the HMD 110 is not necessarily connected to the AP 310 in the room 510 where the HMD 110 is located. For example, depending on the relationship between the arrangement position of the access point 310 and the position where the HMD 110 is located, even when the wearer of the HMD 110 is in the first room 511, it may be connected to the AP 311 installed in the adjacent second room 512. In the present embodiment, even in such a case, the server 700 can appropriately identify the room 510 where the HMD 110 is located.

[0194] The hardware configuration of the HMD 110 in the present embodiment is the same as that in the first embodiment. On the other hand, in terms of functions, the HMD 110 in the present embodiment has a configuration different from that in the first embodiment.

[0195] FIG. 16(a) is a functional block diagram of the HMD 110 in the present embodiment. The HMD 110 further includes an image acquisition unit 217 in addition to the configuration of the first embodiment. The image acquisition unit 217 acquires (captures) an image as visual field information using the camera 170 according to an instruction from the wearer. The acquired image may be a still image or a moving image.

[0196] In addition, when the space shape information acquisition unit 214 in the present embodiment does not have the space shape information of its own room in the HMD-side library 221, it requests the server 700. In the present embodiment, at this time, an image captured in its own room is further transmitted as visual field information.

[0197] Next, the server 700 will be described. The server 700 in the present embodiment basically has the same configuration as that in the first embodiment. However, the processing of the space shape information management unit 721 and the data held by the server-side library 730 in the present embodiment are different.

[0198] First, the server-side library 730 of this embodiment further holds position information of each room 510 or access point 310 in addition to Fig. 16(b). Here, for example, a case where position information of a predetermined position (e.g., southwest corner, etc.) of the room 510 is held will be described as an example. The position information may be, for example, latitude and longitude information, or information based on a coordinate system defined within the system. Also, if the room is polygonal, latitude and longitude information of each corner of the room may be held.

[0199] The spatial shape information management unit 721 receives an image as field of view information together with the spatial shape information request. Then, it determines whether or not the spatial shape information of the room 510 specified by the access point ID received at the same time has an area that matches the field of view information. If not, it determines whether or not there is a matching area for the spatial shape information of other rooms 510 in order of proximity to the room 510. Then, it transmits the spatial shape information of the room that has a matching area to the transmission source. Note that if there is no matching area for the spatial shape information of the rooms 510 within a predetermined range, it sends a reply indicating that the spatial shape information is not registered.

[0200] The spatial shape information transmitted from the HMD 110 in response to the reply is registered in the server-side library 730 in association with the access point ID of the access point 310 that relayed the information.

[0201] The flow of spatial shape information acquisition processing and spatial shape information management processing of this embodiment will be described. The spatial shape information acquisition processing on the HMD 110 side is the same as in the first embodiment. The authentication processing is also the same. Here, the spatial shape information management processing on the server 700 side will be described. FIG. 17 is a flow chart showing the processing flow on the server 700 side of this embodiment. This processing is started when a spatial shape information request is received together with field of view information from the HMD 110.

[0202] The spatial shape information management unit 721 determines whether there is a region in the spatial shape information 733 (room spatial shape information) held in the server-side library 730 that matches the visual field information, in association with the access point ID of the transmission source (step S2101).

[0203] If there is a matching region (S2101; Yes), the spatial shape information management unit 721 returns the spatial shape information to the transmission source (step S2102) and ends the process.

[0204] If there is no matching region (S2101; No), the spatial shape information management unit 721 searches for a matching room 510 (step S2103). Here, first, the distance between the position information of the room 510 associated with the access point ID of the transmission source and the position information of other rooms 510 is calculated, and the same determination as in step S2101 is made in order from the closest room 510. Note that the upper limit M (M is an integer of 1 or more) of the number of rooms to be determined is set in advance.

[0205] The spatial shape information management unit 721 determines whether there is spatial shape information having a matching region from the search result of step S2103 (step S2104). If there is a matching room (S2104; Yes), the spatial shape information is returned to the HMD 110 of the transmission source (step S2102) and the process ends. If not (S2104; No), it is returned to that effect (step S2105) and the process ends.

[0206] In the case of this embodiment, when the HMD 110 receives a reply from the server 700 that the spatial shape information for the room 510 is not held on the server side, as in the first embodiment, the HMD 110 maps the room 510 by itself and generates spatial shape information. Then, it transmits the information for registration to the server 700. At this time, in this embodiment, the access point 310 arranged in the room 510 is intentionally selected and transmitted.

[0207] When the spatial shape information management unit 721 receives the transmission of the spatial shape information from the HMD 110, it registers it in the server-side library 730 in association with the access point ID of the transmission source access point.

[0208] According to this embodiment, not only the access point ID of the access point but also the image acquired by the HMD 110 is used to identify the existence area of the HMD 110.

[0209] According to this embodiment, even when there is a possibility that the HMD 110 is connected to an access point 310 outside the room 510 where the wearer of the HMD 110 exists, the HMD 110 can appropriately acquire and use the spatial shape information stored in the server 700. And at this time, special authentication processing is not required.

[0210] In addition, when the spatial shape information is stored inside the HMD 110, by transmitting the stored spatial shape information to the server 700, the server 700 can compare the spatial shape information and confirm whether it is the corresponding room.

[0211] <Modification Example 5> As described above, in each of the embodiments and modification examples, the GPS (Global Positioning System) is not used. However, the GPS information acquired by the GPS receiver 151 of the HMD 110 may be used.

[0212] For example, in the case of a room in a single-story building, the room 510 where the HMD 110 exists can be identified by the GPS information. As an example, when the shape of the room is rectangular, the latitudes and longitudes of the four corners of the room are registered in the server-side library 730 of the server 700 as the position information of each room.

[0213] By comparing the GPS information (latitude and longitude) transmitted from the HMD 110 with the position information of each room registered in the server 700, the room where the HMD 110 exists is identified.

[0214] <Modification Example 6> The server-side library 730 may register the registration date and time of the spatial shape information. When checking whether the spatial shape information is registered or not, not only the presence or absence but also the registration date and time may be taken into account for judgment. That is, if the difference between the registration date and time and the current date and time is equal to or more than a predetermined value, even if it is registered, it is judged as unregistered.

[0215] For example, the state in the room 510 changes. Thus, by managing the spatial shape information by date and time, it is possible to cope with changes in the interior arrangement and the like.

[0216] <Modification Example 7> In addition, in each of the above embodiments, basically, the relay device to the communication network arranged in each room 510 is used as the identification information of each room. However, the identification information is not limited to this. For example, when each room is equipped with a unique device capable of transmitting information for specifying the room, the device may be used. Examples of such a device include, for example, a room entry / exit management device with a communication function.

[0217] Furthermore, in each of the above embodiments, the case where the HMD 110 is used as the portable information terminal has been described as an example, but the portable information terminal is not limited to this. For example, it may be various portable information terminals having the same functions such as a notebook PC or a tablet PC.

[0218] Also, the configuration for realizing the present invention is not limited to these embodiments and modification examples, and various modification examples can be considered. For example, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can also be added to the configuration of one embodiment. All of these belong to the scope of the present invention. Also, the numerical values, messages, etc. appearing in the text and drawings are merely examples, and using different ones will not impair the effects of the present invention.

[0219] Note that the functions and the like of each of the above-described embodiments may be realized in hardware by designing a part or all of them, for example, by means of an integrated circuit. Alternatively, they may be realized in software by a microprocessor unit or the like interpreting and executing a program that realizes each function and the like. A combination of hardware and software may also be used. The software may be stored in advance in the ROM 142 of the HMD 110 or the like at the time of product shipment. After product shipment, it may be acquired from various server devices on the Internet or the like. Alternatively, it may be provided by being stored in a storage medium such as a memory card or an optical disk.

[0220] In addition, the control lines and information lines shown in the drawings indicate those considered necessary for explanation, and do not necessarily show all the control lines and information lines on the product. In reality, it may be considered that almost all components are interconnected.

Explanation of Reference Numerals

[0221] 100: Presence Area Identification System, 100a: Presence Area Identification System, 110: HMD, 111: First HMD, 112: Second HMD, 113: System Bus, 120: Main Processor, 141: RAM, 142: ROM, 143: Flash Memory, 151: GPS Receiver, 152: Geomagnetic Sensor, 153: Distance Sensor, 154: Acceleration Sensor, 155: Gyro Sensor, 156: Timer, 161: Wireless Communication I / F, 162: Telephone Network Communication I / F, 170: Camera, 171: In-Camera, 172: Out-Camera, 173: Display, 181: Microphone, 182: Speaker, 183: Audio Decoder, 191: Button Switch, 192: Touch Panel 211: Main Control Unit, 212: Authentication Unit, 213: Data Management Unit, 214: Spatial Shape Information Acquisition Unit, 215: AR Image Processing Unit, 216: Communication Unit, 217: Image Acquisition Unit, 221: HMD-Side Library, 221a: Access Point ID, 221b: Spatial Shape Information, 221c: AR Image Information, 221d: BS-ID, 221e: Resolution, 222: Authentication Information Table 310: Access Point, 311: First Access Point, 312: Second Access Point, 321: First BS, 322: Second BS, 330: Access Point, 331: Access Point, 332: Access Point, 340: Hub, 510: Room, 511: First Room, 512: Second Room, 520: Table, 610: Network, 620: Mobile Phone Line Network, 621: Mobile Phone Line Network, 622: Mobile Phone Line Network, 630: Network Network, 631: Mobile Management, 632: Home Subscriber Server, 700: Server, 720: Main Processor, 721: Spatial Shape Information Management Department, 730: Server - side Library, 731: Room Number, 732: Access Point ID, 733: Spatial Shape Information, 734: Resolution, 735: AR Image Information, 736: Resolution, 741: RAM, 742: ROM, 743: Flash Memory, 761: Communication I / F, 773: Display, 791: Operating Device, T421: Time Axis, T422: Time Axis, T423: Time Axis, T424: Time Axis, T451: Time Axis, T452: Time Axis, T453: Time Axis

Claims

1. A portable information terminal, A display and A communication interface for connecting to a server via a relay device; an input unit into which instructions from a user of the portable information terminal are input; Memory, A processor; Equipped with The processor, Acquire relay device identification information, which is information for identifying a plurality of relay devices, from a plurality of relay devices arranged in a space in which the portable information terminal exists; selecting a relay device to be used for connecting to the server from among the plurality of relay devices based on an instruction from the user inputted by the input unit or a priority order stored in the memory; reading from the memory a password stored in association with the selected relay device; Connecting to the selected relay device using the read password; if spatial shape information that maps the space required for displaying the space on the display is not registered in the memory, a request for the spatial shape information is made to the server via the selected relay device by the communication interface; receiving the spatial shape information transmitted from the server in response to the request; storing the received spatial shape information in the memory in association with the relay device identification information of the selected relay device; A portable information terminal characterized by:

2. 2. The portable information terminal according to claim 1, a camera or a distance sensor; When the processor does not receive the spatial shape information from the server, the processor generates shape information of the space using the camera or the distance sensor, and stores the generated shape information in the memory in association with the relay device identification information of the selected relay device. A portable information terminal characterized by:

3. 3. The portable information terminal according to claim 1, The relay device is an access point of a wireless LAN. A portable information terminal characterized by:

4. 3. The portable information terminal according to claim 1, The communication between the communication interface and the relay device uses millimeter waves. A portable information terminal characterized by:

5. 3. The portable information terminal according to claim 1, The relay device is a mobile phone base station. A portable information terminal characterized by:

6. 3. The portable information terminal according to claim 1, The relay device is a wired LAN relay device. A portable information terminal characterized by:

7. 2. The portable information terminal according to claim 1, a camera or a distance sensor; The memory stores a best resolution that is a best resolution of the shape information of the space that can be generated by the processor; The processor, When the spatial shape information is received from the server, a resolution of the received spatial shape information is compared with the best resolution; If the best resolution is higher, generating shape information of the space at the best resolution using the camera or the distance sensor; storing the generated shape information in the memory in association with the relay device identification information of the selected relay device, and transmitting the generated shape information together with information indicating the best resolution to the server via the selected relay device. A portable information terminal characterized by:

8. A portable information terminal; A plurality of relay devices; A server; An information processing system comprising: The portable information terminal includes: a display, a communication interface for connecting to a server, an input unit for inputting an instruction from a user of the portable information terminal, a memory, and a processor; The processor, acquiring relay device identification information, which is information for identifying the plurality of relay devices, from the plurality of relay devices; selecting a relay device to be used for connecting to the server from among the plurality of relay devices based on an instruction from the user inputted by the input unit or a priority order stored in the memory; reading from the memory a password stored in association with the selected relay device; Connecting to the selected relay device using the read password; if spatial shape information that maps a space required for displaying the display in the space where the portable information terminal exists is not registered in the memory, a request for the spatial shape information is made to the server via the selected relay device by the communication interface; receiving the spatial shape information transmitted from the server in response to the request; storing the received spatial shape information in the memory in association with the relay device identification information of the selected relay device; An information processing system comprising:

9. 9. The information processing system according to claim 8, At least one of the relay devices is provided in each of a plurality of spaces separated by structures. An information processing system comprising:

10. 9. The information processing system according to claim 8, the selected relay device transmits the relay device identification information of the selected relay device to the server when making the request for the spatial shape information; The server includes a server storage device and a server processor; the server storage device registers the spatial shape information of the space in which the selected relay device is placed in association with the relay device identification information of the selected relay device; When the server processor receives the request from the portable information terminal via the selected relay device, the server processor transmits the corresponding spatial shape information to the portable information terminal based on the received relay device identification information. An information processing system comprising:

11. 11. The information processing system according to claim 10, The plurality of relay devices are a first relay device and a second relay device having different performances, In the server storage device, different pieces of spatial shape information are registered in association with the relay device identification information. An information processing system comprising:

Images