Portable information terminal, wirelessly connectable external device, and information processing method therefor

Abstract

Provided are a portable information terminal, an external device capable of wireless connection, and an information processing method therefor, whereby the location of the external device capable of wireless connection can be determined, and an external device to which a user wants to connect can be visually selected. To this end, the portable information terminal capable of being wirelessly connected to an external device has: a first communication function and a second communication function capable of being wirelessly connected to the external device; a camera that captures an image of the surroundings of the portable information terminal; an image processing unit that processes an image captured by the camera; a display for displaying the image processed by the image processing unit; a storage unit that stores first identification information of the external device used in the first communication function and second identification information of the external device used in the second communication function; and a control unit that calculates the relative position of the external device with respect to the portable information terminal using the second communication function, and the image processing unit generates a superimposed image in which the first identification information is superimposed on the calculated relative position of the external device in the image, and displays the superimposed image on the display.

Description

Technical Field

[0001] The present invention relates to a portable information terminal that provides an external device selection assistance method for selecting an external device to which the portable information terminal can be wirelessly connected. Background Art

[0002] In recent years, data communication is generally performed wirelessly for digital devices at short distances. In particular, wireless communication based on the Bluetooth (registered trademark) standard has expanded its application in external devices including speakers, headphones, etc. in addition to information terminals such as personal computers and smart phones. Here, broadcast communication is generally used to search for external devices that can be connected wirelessly. However, even if it is possible to detect external devices that can be connected wirelessly, it is difficult to accurately determine their location, that is, both the direction and the distance.

[0003] On the other hand, portable information terminals equipped with UWB (Ultra Wide Band) communication functions, which are ultra-wideband wireless communications, have begun to appear. By utilizing this UWB communication function, position detection can be performed with high accuracy. Generally speaking, the position accuracy based on the Bluetooth communication function is in meters, while the position accuracy based on the UWB communication function is called centimeters.

[0004] As a background art in this technical field, there is Patent Document 1. Patent Document 1 discloses that an external wireless terminal having a Bluetooth communication function adds location information of the external wireless terminal to device name information to be transmitted during scanning processing and transmits the device name information.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2013-247428 Summary of the invention

[0008] Problems to be solved by the invention

[0009] Patent document 1 has the following problems: the external wireless terminal side needs to obtain the location information of the external wireless terminal, and the location information is based on GPS (Global Positioning System) information, so it cannot be used in places where GPS radio waves cannot be received, such as indoors. In addition, Patent document 1 shows an example of displaying the relative position of the external wireless terminal on the screen, but the user needs to judge whether the external wireless terminal displayed on the screen is consistent with the actual external wireless terminal. Therefore, the action of the user selecting the external wireless terminal to be connected is not intuitive, and the user may make a mistake in selecting the external wireless terminal.

[0010] The purpose of the present invention is to take the above-mentioned problems into consideration and provide a portable information terminal that can grasp the positions of multiple external devices with Bluetooth communication function, intuitively determine the external device to be connected from the multiple external devices with Bluetooth communication function according to the grasped positions of the external devices, the external device that can be connected wirelessly, and the information processing method therefor.

[0011] Means for solving problems

[0012] If one example is cited, the present invention is a portable information terminal that can be wirelessly connected to an external device, and has: a first communication function and a second communication function that can be wirelessly connected to an external device; a camera that captures the surroundings of the portable information terminal; an image processing unit that processes the image captured by the camera; a display that displays the image processed by the image processing unit; a storage unit that stores first identification information of an external device used in the first communication function and second identification information of the external device used in the second communication function; and a control unit, the control unit uses the second communication function to calculate the relative position of the external device with respect to the portable information terminal, and the image processing unit generates an overlapping image in which the first identification information is overlapped at the calculated relative position of the external device in the image, and displays it on the display.

[0013] Effects of the Invention

[0014] According to the present invention, by superimposing the external device identification information on the external device image extracted from the photographic image, the appearance of the external device and its identification information can be easily identified and matched, thereby enabling the user to intuitively select the external device to be connected. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram for explaining the outline of the entire system in Example 1.

[0016] Figure 2 This is an external view of the HMD in Example 1.

[0017] Figure 3 This is a block diagram of the hardware structure of the HMD in the first embodiment.

[0018] Figure 4 This is a functional block diagram of the HMD in the first embodiment.

[0019] Figure 5 This is a flowchart of the external device selection auxiliary processing in the first embodiment.

[0020] Figure 6 This is a flowchart of the advertisement / scanning process of the HMD in the first embodiment.

[0021] Figure 7This is a flowchart of the UWB connection process of the HMD in the first embodiment.

[0022] Figure 8 This is a block diagram of the main hardware structure of the external device suitable for Example 1.

[0023] Fig. 9 This is a flowchart of the advertising / scanning process of the external device suitable for Example 1.

[0024] Fig.10 This is a flowchart of the UWB connection process of the external device suitable for the first embodiment.

[0025] Fig.11 This is an external device information table storing the external device information in the first embodiment.

[0026] Fig.12 This is a schematic diagram for explaining the calculation principle of the external device position in Example 1.

[0027] Fig.13 This is a schematic diagram showing an example of displaying the external device name in the first embodiment.

[0028] Fig.14A This is a schematic diagram showing an example of displaying the external device name in the second embodiment.

[0029] Fig. 14B This is a schematic diagram showing an example of displaying the external device name in the second embodiment.

[0030] Fig. 14C This is a schematic diagram showing an example of displaying the external device name in the second embodiment.

[0031] Fig.14D This is a schematic diagram showing an example of displaying the external device name in the second embodiment.

[0032] Fig.15A This is a schematic diagram showing an example of displaying the positions and device names of external devices in the third embodiment.

[0033] Fig. 15B This is a schematic diagram showing an example of displaying the positions and device names of external devices in the third embodiment.

[0034] Fig.16 This is a schematic diagram showing an example of displaying the external device name in the fourth embodiment.

[0035] Fig.17 This is a schematic diagram showing an example of displaying the external device name in the fifth embodiment.

[0036] Fig.18 This is a schematic diagram for explaining the outline of the entire system in Example 6.

[0037] Fig.19 This is an appearance diagram of the portable information terminal in Example 6.

[0038] Fig. 20 This is a flowchart of the external device selection auxiliary processing in the seventh embodiment.

[0039] Fig.21 This is an external device information table storing the external device information in the seventh embodiment. DETAILED DESCRIPTION

[0040] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0041] Example 1

[0042] Figure 1 This is a schematic diagram for explaining the overall outline of the system in this embodiment. Figure 1 , the following situation is shown: the user 10 wears a glass-shaped see-through HMD (Head Mounted Display) 1 as a portable information terminal, and the user 10 searches whether an external device that can communicate with the HMD 1 through Bluetooth exists in the field of view of the HMD 1. The details of the search process will be described later, but the HMD 1 searches whether the desktop PC 301 in the field of view can communicate with Bluetooth as an external device. In addition, the HMD 1 is connected to a network 17 connected to a network server 16 via an access point 15. The network server 16 includes a network server that performs various calculations, a network server that stores various data, etc., and the HMD 1 can flexibly use it as needed.

[0043] Figure 2 : is an appearance diagram of the HMD in this embodiment. Figure 2 In the embodiment, HMD1 has a display screen 75 at the left and right lens positions of the glasses, a right camera 711 is arranged at the right end of the right lens position of the glasses, and a left camera 712 is arranged at the left end of the left lens of the glasses. HMD1 can visually confirm the actual space through the transparent display screen 75. In addition, it is possible to display an augmented reality AR (Augmented Reality) object on the display screen 75. Therefore, the user 10 wearing HMD1 can visually confirm both the augmented reality AR object displayed on the display screen 75 and the status of the actual space at the same time. In addition, the speaker is configured with a right speaker 821 and a left speaker 822 at the portion corresponding to the temple of the glasses.

[0044] In addition, HMD1 has a UWB communication function and can measure the distance to a connected device having a UWB communication function. The UWB communication transmitting and receiving antennas include a right UWB antenna 641 at the rightmost lower part of HMD1, a left UWB antenna 642 at the leftmost lower part of HMD1, and a center UWB antenna 643 at the center of HMD1.

[0045] In the following, the specific structure and method in this embodiment are described. In addition, in this embodiment and the drawings, the same symbols are attached to the same functions for description.

[0046] Figure 3 : is a hardware structure block diagram of the HMD in this embodiment. Figure 3 In the present invention, the HMD 1 includes a main control unit 2, a system bus 3, a storage unit 4, a sensor unit 5, a communication processing unit 6, an image processing unit 7, a sound processing unit 8, and an operation input unit 9.

[0047] The main control unit 2 is a microprocessor unit that controls the entire HMD 1 according to a predetermined operation program. The system bus 3 is a data communication path for transmitting and receiving various commands, data, etc. between the main control unit 2 and each component block in the HMD 1.

[0048] The storage unit 4 is composed of a program unit 41 storing programs for controlling the actions of the HMD 1, a various data unit 42 storing various data such as action setting values, detection values ​​from the sensor unit, objects including content, and library information downloaded from the library, and a rewritable program function unit 43 such as a work area used in various program actions. In addition, the storage unit 4 can store action programs downloaded from the network, various data created by the action programs, etc. In addition, it can also store content such as dynamic images, static images, and sounds downloaded from the network. In addition, data such as dynamic images and static images captured by a camera can be stored. In addition, the storage unit 4 can store necessary information (setting values ​​such as thresholds, image data, etc.) in advance. The storage unit 4 needs to maintain the stored information even when the power of the HMD 1 is turned off, and uses semiconductor device memories such as flash ROM, SSD (Solid State Drive), and disk drives. In addition, each action program stored in the storage unit 4 can be updated and expanded in function by downloading from each server device on the network.

[0049] The sensor unit 5 is a group of various sensors for detecting the state of the HMD 1 and the surroundings. The sensor unit 5 is composed of a GPS receiving unit 51, a geomagnetic sensor 52, an acceleration sensor 53, a gyro sensor 54, and a distance sensor 55. These sensor groups can detect the position, tilt, orientation, movement, etc. of the HMD 1. In addition, the HMD 1 may also have other sensors such as an illumination sensor and an altitude sensor.

[0050] The communication processing unit 6 is composed of a LAN (Local Area Network) communication unit 61, a telephone network communication unit 62, a Bluetooth communication unit 63, and a UWB communication unit 64. The LAN communication unit 61 is connected to a network 17 such as the Internet via an access point 15, etc., and transmits and receives data with each network server 16 on the network 17. The connection with the access point 15, etc. is performed by a wireless connection such as Wi-Fi (registered trademark). The telephone network communication unit 62 performs telephone communication (calls) and data transmission and reception through wireless communication with a base station of a mobile phone communication network, etc. The communication with the base station, etc. can be performed by LTE (Long Term Evolution) mode, 5G mode (a fifth-generation mobile communication system with high speed, large capacity, low latency, and multiple simultaneous connections as the goal) or other communication modes. The Bluetooth communication unit 63 performs Bluetooth communication such as advertising / scanning processing and data communication described later with an external device having a Bluetooth communication function. The UWB communication unit 64 performs UWB communication such as UWB connection processing and UWB data communication with an external device having a UWB communication function. The LAN communication unit 61, the telephone network communication unit 62, the Bluetooth communication unit 63, and the UWB communication unit 64 each have a coding circuit, a decoding circuit, an antenna, etc. In particular, the UWB antenna in the UWB communication unit is an important component related to positioning, and the configuration of the three UWB antennas is a configuration that takes positioning into consideration. In addition, the communication processing unit 6 may also have other communication units such as an infrared communication unit.

[0051] The image processing unit 7 is composed of a camera unit 71 and a display unit 72. The camera unit 71 is a camera that converts light input from a lens into an electrical signal through an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor, thereby obtaining image data of the surroundings and objects. In this embodiment, it is composed of a right camera 711 and a left camera 712. The display unit 72 is a display device such as a transmissive display based on a semi-transparent liquid crystal, which constitutes a display screen 75 and displays AR objects, accompanying information, etc. to the user 10 of the HMD 1.

[0052] The sound processing unit 8 is composed of a sound input unit 81 and a sound output unit 82. The sound input unit 81 is a microphone that converts the sound of the actual space, the user's voice, etc. into sound data. For example, microphones are respectively arranged near the right camera 711 and the left camera 712. The sound output unit 82 is a speaker that outputs sound information required by the user. Of course, earphones or headphones can also be connected and used according to the purpose. The operation input unit 9 is an operation input unit for inputting operation instructions for the HMD 1.

[0053] also, Figure 3 The hardware configuration example of the HMD 1 shown also includes a plurality of configurations that are not essential to the present embodiment, but even without these configurations, the effects of the present embodiment are not impaired. In addition, configurations not shown, such as an electronic money settlement function, may be further added.

[0054] Figure 4 : is a functional block diagram of the HMD in this embodiment. Figure 4 Among them, the control 21 is a function of controlling the entire HMD 1 , and is mainly processed by the main control unit 2 , the program unit 41 of the storage unit 4 , and the program function unit 43 .

[0055] The Bluetooth connection process 22 is a function of performing an advertisement / scan process with an external device having a Bluetooth communication function using the Bluetooth communication unit 63 of the communication processing unit 6. The Bluetooth device information storage 23 is a function of storing the device information of the external device obtained in the advertisement / scan process in the Bluetooth connection process 22 in the various data unit 42 of the storage unit 4.

[0056] The UWB connection process 24 is a function of performing polling / response / message processing with an external UWB device having a UWB communication function using the UWB communication unit 64 of the communication processing unit 6. The UWB device information storage 25 is a function of storing the device information of the external UWB device obtained in the polling / response / message processing in the UWB connection process 24 in the various data units 42 of the storage unit 4. The device position calculation process 26 is a function of performing transmission and reception between the HMD1 and the external UWB device having a UWB communication function through the UWB connection process 24, and calculating the distance based on the elapsed time of the transmission and reception. The specific method will be described later, and the distance from the right UWB antenna 641, the left UWB antenna 642, and the center UWB antenna 643 to the external device is calculated respectively, and the relative position of the HMD1 and the external UWB device is calculated. The device position information storage 27 is a function of storing the relative position information between the HMD1 and the external device calculated by the device position calculation process 26 in the various data units 42 of the storage unit 4.

[0057] The image data acquisition 28 is a function of acquiring image data by the camera unit 71 of the image processing unit 7 capturing the vicinity of the relative positions of the HMD 1 and the external device stored in the device position information storage 27. The device image extraction 29 is a function of analyzing the image data acquired by the image data acquisition 28 to extract the image of the external device.

[0058] The device name superimposed output 30 is a function of superimposing the device name in the device information of the external device stored in the Bluetooth device information storage 23 on the display unit 72 of the image processing unit 7 by an AR object. Specifically, the device name in the device information of the external device stored in the Bluetooth device information storage 23 is superimposed and displayed by an AR object near the relative position of the HMD 1 and the external device stored in the device position information storage 27.

[0059] The external device selection process 31 is a function of selecting an external device connected by the user 10 through the operation of the operation input unit 9 from among the external device names superimposed and displayed by the device name superimposed output 30 .

[0060] In this embodiment, the characteristic functions are realized by external devices communicating with the HMD 1. Examples of the external devices include Figure 1 The desktop PC 301 is shown. Figure 8 901 is a block diagram of the main hardware structure of the external device 901 suitable for this embodiment. Figure 8 In the basic structure and Figure 3 The HMD 1 of the present embodiment shown in the figure is the same as that of the HMD 1 of the present embodiment, and the main control unit 902, the system bus 903, the storage unit 904, and the communication processing unit 906 are used as the basic structure suitable for the present embodiment. The storage unit 904 is composed of a program unit 941, various data units 942, and a program function unit 943, and the communication processing unit 906 is composed of a LAN communication unit 961, a telephone network communication unit 962, a Bluetooth communication unit 963, and a UWB communication unit 964. The description of each block is the same as that of the present embodiment. Figure 1 The description is the same as in , and the description here is omitted. Figure 8 Only the configuration required to implement this embodiment is shown, and other configurations for realizing other functions may be added.

[0061] Figure 5 4 is a flowchart showing the process of the external device selection auxiliary processing (S400) in this embodiment. Figure 4 Functional block diagram of Figure 5 The processing process is described below.

[0062] exist Figure 5 When S400 (S401) is started, the advertisement / scanning process (S420) as a subroutine is first executed. The advertisement / scanning process is a process for acquiring device information of an external device through the Bluetooth connection process 22.

[0063] Here, the advertisement / scanning process (S420) as a subroutine will be described. Figure 6 4 is a flowchart showing the processing procedure of the advertisement / scanning process (S420). Figure 4 Functional block diagram of Figure 6 The processing process is described below.

[0064] exist Figure 6 When the process of S420 is started (S421), first, it is determined whether new advertisement data can be received (S422). Advertisement is a stage in which an external device with Bluetooth communication function sends advertisement data to the outside through broadcast communication. The advertisement data includes the device name of the external device.

[0065] If it is determined in S422 that new advertisement data has not been received, the process moves to S423. In S423, a timeout determination is performed, and if it is determined that the predetermined time has not been reached, the process moves to S422 again. In S423, if the predetermined time has been reached, the process of S420 ends (S431).

[0066] If it is determined in the determination process of S422 that new advertisement data can be received, the process is transferred to the scan request transmission process (S424). In addition, if a plurality of new advertisement data can be received, the HMD 1 selects the new advertisement data first detected in the process of polling and searching the external device and executes the process of S424. In the scan request transmission process (S424), a scan request is transmitted from the HMD 1 to the external device that transmits the new advertisement data determined in the determination process of S422.

[0067] Next, the HMD 1 determines whether there is a scan response from the target external device (S425). In S425, if it is determined that there is no scan response from the target external device, the process moves to S426. In S426, a timeout determination is performed, and if the predetermined time has not been reached, the process moves to S425 again. In S426, if the predetermined time has been reached, the process of S420 is terminated (S431). In S425, if it is determined that a scan response can be received from the target external device, the device information of the external device including the device name is saved by the Bluetooth device information storage 23 (S427).

[0068] Next, it is determined whether the information received from the external device contains the UWB device ID (S428). If it is determined in S428 that the information received from the external device contains the UWB device ID, the UWB device ID is saved in the table through the UWB device information storage 25 (S429), and the process is transferred to S430. The details of the table will be described later. If it is determined in S428 that the information received from the external device does not contain the UWB device ID, the process is transferred to S430.

[0069] In S430, it is determined whether new advertisement data can be received. If it is determined in S430 that new advertisement data can be received, the process returns to the scanning request sending process of S424. If it is determined in S430 that new advertisement data cannot be received, there is no new external device, and S420 ends (S431). Figure 6 The process of S420 (advertisement / scanning process) of the subroutine shown.

[0070] Here, return to Figure 5 In Figure 5 In the process, if the advertisement / scanning process (S420) as a subroutine is completed, it transfers to the UWB connection process (S440) as a subroutine.

[0071] Next, the UWB connection process (S440) will be described. Figure 7 It is a flowchart showing the processing procedure of the UWB connection process (S440).

[0072] exist Figure 7 In the process of S440, when the process of S441 is started, first, it is determined whether the UWB polling signal from the UWB external device can be received (S442). In S442, if it is determined that the UWB polling signal from the UWB external device cannot be received, the process moves to S451. In S442, if it is determined that the UWB polling signal from the UWB external device can be received, the process moves to S443. The UWB polling signal from the external device includes the UWB device ID of the external device, and the HMD 1 can obtain the UWB device ID information of the external device.

[0073] S443 is a process for determining whether the acquired UWB device ID of the external device is new. If it is determined in S443 that the acquired UWB device ID of the external device is not new, the process moves to S451. If it is determined in S443 that the acquired UWB device ID of the external device is new, the process moves to S444.

[0074] S444 is a process of storing the acquired UWB device information in a table by the UWB device information storage 25. In S444, the UWB device information storage 25 stores the UWB device ID acquired from the received UWB polling signal of the external device and the reception time of the UWB polling signal in the table.

[0075] Next, a UWB response transmission process of transmitting a UWB response to the UWB external device is performed (S445). Next, the UWB response transmission time, which is the transmission time of the response transmitted to the external UWB device, is stored in a table by the UWB device information storage 25 (S446).

[0076] Next, it is determined whether the UWB message signal from the external device can be received (S447). If it is determined in S447 that the UWB message signal from the external device cannot be received, the process moves to S448. S448 is a timeout determination process. If it is determined that the predetermined time has not been reached, the process moves to S447 again. If it is determined in S448 that the predetermined time has been reached, the process moves to S451. S448 is a process of only waiting for the reception of the UWB message, but it is necessary to avoid deadlock of the system. If it is determined in S447 that the UWB message signal from the external device can be received, the process moves to S449.

[0077] S449 is a UWB device information storage process, which is a process of storing the reception time of the UWB message received in the HMD 1 and the UWB device information (UWB polling transmission time, UWB response reception time and UWB message transmission time) acquired by the UWB message signal in a table through the UWB device information storage 25 .

[0078] Next, the relative distance to the external device is calculated, and the calculated relative distance is stored in the table by the device position information storage 27 (S450). The detailed process of distance calculation will be described later.

[0079] Next, the process moves to S451. S451 is a timeout determination process. If it is determined that there is no timeout, the process returns to S442. If it is determined that there is a timeout in S451, the process of S440 ends (S452). In the timeout determination process of S451, a time that is sufficiently longer than the generation cycle of the UWB polling signal is set. In addition, the user 10 can also terminate it arbitrarily. The above is Figure 7 The subroutine shown is the process of S440 (UWB connection process).

[0080] In order to realize its functions, the HMD 1 of this embodiment needs to be connected to an external device suitable for this embodiment. Therefore, the processing of the external device suitable for this embodiment, especially the advertisement / scanning processing, will be described. Fig. 9 This is an example of a flowchart showing a processing procedure of an advertisement / scanning process in an external device suitable for this embodiment.

[0081] exist Fig. 9In the process, when the processing starts (S521), the advertisement data is sent through broadcast communication (S522). Next, it is determined whether a scan request is received (S523). If it is determined in S523 that the scan request is not received, the process moves to S524. S524 is a timeout determination process. If the predetermined time is not reached, the process moves to S522 again. If it is determined in S524 that the predetermined time is reached, the process ends (S528). If it is determined in S523 that the scan request can be received, the process moves to S526.

[0082] In S526 , the UWB device ID is added to the scan response, and in S527 , the scan response to which the UWB device ID is added is transmitted to the HMD 1 , and the process ends ( S528 ).

[0083] Here, the UWB connection processing in the external device assumed in this embodiment is described. Fig.10 This is an example of a flowchart showing a processing procedure of a UWB connection process in an external device suitable for this embodiment.

[0084] exist Fig.10 In the process, when the process starts (S541), a UWB polling signal is sent through broadcast communication (S542). At this time, the time when the UWB polling signal is sent is also saved (S543).

[0085] Next, it is determined whether a UWB response signal from HMD1 can be received (S544). If it is determined in S544 that a UWB response signal from HMD1 cannot be received, the process proceeds to S550. If it is determined in S544 that a UWB response signal from HMD1 can be received, the time at which the UWB response signal is received is stored (S545).

[0086] Next, in S547, the device name is added to the UWB message information. Next, in S548, the time when the UWB message signal is transmitted is stored. In S549, the UWB message signal to which the UWB message storage time stored in S548 is added is transmitted.

[0087] Next, the process moves to a determination process of whether or not there is an end instruction (S550). If it is determined in S550 that there is no end instruction, the process returns to S542. If there is an end instruction in S550, the UWB connection process in the external device is terminated (S551).

[0088] As explained above, in Fig. 9 In the advertising / scanning process, the UWB device ID is added to the scan response. Fig.10In the UWB connection process of the embodiment, the device name is added to the UWB message information. The external device suitable for this embodiment can adopt at least one of the process of adding the UWB device ID to the scan response and the process of adding the device name to the UWB message information. That is, the HMD side can obtain the device name or UWB device ID from the external device according to any of the processes in the external device. In addition, which process to adopt can be determined according to the specifications of the external device.

[0089] Here, the external device information table storing information in the Bluetooth device information storage 23 and the UWB device information storage 25 will be described. Fig.11 This is an example of the external device information table T800.

[0090] The external device information table T800 is configured by associating Bluetooth related information and UWB related information for each external device. The external device number is a number assigned by the HMD 1 to identify each external device.

[0091] Bluetooth related information consists of the device name indicating the device name, the service UUID (Universally Unique Identifier) ​​indicating the function, the company identifier, the device address inherent to the device, etc. UWB related information consists of the UWB device ID, polling reception time Trp, response transmission time Tsr, message reception time Trf, polling transmission time Tsp, UWB response reception time Trr, UWB message transmission time Tsf, and the distance Da, Db, and Dc from each antenna to the external device.

[0092] Here, a method for calculating the distance to the external device is described. The distance to the external device is calculated by the UWB signal arrival time. Considering that the time system of HMD1 and the time system of the external device are not synchronized (not the same time system), the UWB signal arrival time T can be calculated by the following formula (1).

[0093] T={(Trr-Tsp)-(Tsr-Trp)+

[0094] (Trf-Tsr)-(Tsf-Trr)} / 4…(1)

[0095] Furthermore, if the UWB signal arrival time T can be calculated, the transmission speed of the UWB signal is the speed of light c, and therefore the distance D to the external device can be calculated by the following equation (2).

[0096] D=T×c…(2)

[0097] The distance D to the external device is the relative distance from the UWB antenna of HMD 1. Therefore, when HMD 1 has three UWB antennas (right UWB antenna 641, left UWB antenna 642, and center UWB antenna 643), the distance Da is calculated using the right UWB antenna 641, the distance Db is calculated using the left UWB antenna 642, and the distance Dc is calculated using the center UWB antenna 643, and stored in table T800.

[0098] Here, the calculation principle of calculating the relative position of the external device with respect to the HMD 1 based on the distance D from each UWB antenna to the external device will be described. Fig.12 1 is a schematic diagram showing the calculation principle for calculating the relative position between the HMD 1 and the UWB device. The HMD 1 has three UWB antennas: a right UWB antenna 641 , a center UWB antenna 643 , and a left UWB antenna 642 .

[0099] The intersection 745 of the circle 731 of the straight-line distance 721 from the right UWB antenna 641 to the external UWB device, the circle 733 of the straight-line distance 723 from the center UWB antenna 643 to the external UWB device, and the circle 732 of the straight-line distance 722 from the left UWB antenna 642 to the external USB device is the position of the external UWB device to be found. Specifically, it is the position of the distance 742 to the left from the position of the center UWB antenna 643 and the distance 743 to the front of the HMD 1.

[0100] In addition, the angle 744 relative to the position of the center UWB antenna 643 can be calculated by the following equation (3).

[0101] Angle 744 = arctan (distance 743 / distance 742) ... (3)

[0102] Through the above calculation principle, the relative position of the UWB device with respect to the HMD 1 can be calculated.

[0103] The obtained information on the relative position of the UWB device with respect to the HMD 1 is stored in the device position information storage 27. In addition, the protocol related to UWB is not limited to the protocol described in this embodiment, and may be other protocols.

[0104] Here, return to Figure 5 The description continues. When the UWB connection process (S440) as a subroutine is completed, the process moves to the external device selection process (S402) for selecting an external device that can be connected. In the external device selection process (S402), the external device obtained in the advertisement / scanning process (S420) is selected. Specifically, the external device is selected in the device number order of the external device information table T800.

[0105] Next, determine whether the relative position of the selected external device exists in the field of view of HMD1 (S403). Specifically, the relative position of the selected external device is determined uniquely based on the distance in the left and right directions and the front distance, or the straight-line distance to the external device and its direction (front angle) with the position of the central UWB antenna 643 in HMD1 as the starting point. In the case where it is determined in S403 that the relative position of the selected external device does not exist in the field of view of HMD1, transfer to S409. In the case where it is determined in S403 that the relative position of the selected external device exists in the field of view of HMD1, photography is performed by acquiring photography data 28 (S404). At this time, enlarged photography can also be limited to the vicinity of the relative position of the external device. The purpose of the photography processing in S404 is to capture an image containing the external device.

[0106] Next, the image of the external device is extracted from the photographic image captured in S404 by the device image extraction 29 (S405). As an example, an object existing in a relative position of the external device can be extracted using image recognition technology. Alternatively, the appearance information of the external device can be obtained from a server connected via a network. In this case, by sending the Bluetooth device name and service UUID stored by the Bluetooth device information storage 23 to the server, the appearance information of the external device can be searched through the server.

[0107] Next, it is determined whether the image of the external device can be extracted (S406). If it is determined in S406 that the image of the external device cannot be extracted, the process proceeds to S409. If it is determined in S406 that the image of the external device can be extracted, the process proceeds to S407.

[0108] In S407, it is determined whether the position of the extracted external device image is consistent with the relative position of the external device detected in the UWB connection process (S440). If it is determined in S407 that the position of the external device image is consistent with the relative position, transfer to S408. Since the extracted external device image has a certain area, it is sufficient to determine whether the relative position is included in the range. In addition, it is also possible to consider the error of the detected relative position information and expand the area for consistency judgment. In addition, the distance sensor 55 can also be used to measure the distance of the area corresponding to the extracted external device image to determine whether the distance is also consistent. If it is determined in S407 that it is inconsistent, transfer to S409.

[0109] S408 is a process of generating an image in which the device name as Bluetooth device information is superimposed on the position of the external device on the photographed image by the device name superimposition output 30, and displaying it by the image processing unit 7.

[0110] here, Fig.13This shows a display example where external device names are displayed in an overlapping manner. Fig.13 The screen shown is a screen that the user 10 wearing the HMD 1 visually checks through the transmissive display screen 75. Fig.13 In the example, the desktop PC 301 and the HMD 302 are present as external devices in the field of view of the HMD 1. At this time, if the desktop PC 301 is an external device suitable for this embodiment, a frame line 303 is superimposed on the position of the desktop PC 301 in the dialog box display, in which "PC MX" is indicated as the name of the external device. Similarly, if the HMD 302 is an external device suitable for this embodiment, a frame line 304 is superimposed on the position of the HMD 302 in the dialog box display, in which "HMD-100" is indicated as the name of the external device. In this way, by superimposing the device name on the position of the external device in the field of view, the user 10 can intuitively determine the device that can be connected.

[0111] Here, return to Figure 5 Continuing the description. After the external device name overlapping display processing is performed in S408, it is determined whether the processing of all detected external devices has been completed (S409). If it is determined in S409 that the processing of all detected external devices has not been completed, return to S402. If it is determined in S409 that the processing of all detected external devices has been completed, transfer to S410.

[0112] S410 is a process of selecting an external device that the user 10 wants to connect to through the external device selection process 31. Fig.13 As shown, this is achieved by displaying a connection cursor 305, moving the cursor position to the dialog box display frame line 303 or 304, and indicating a selection. The movement of the connection cursor can be performed, for example, by gesture detection or line of sight detection of the user 10. In addition, it can also be set as a sound-based selection. Specifically, the sound input from the sound input unit 81 of the sound processing unit 8 is recognized, and the selection is made using the result of the sound recognition. For example, when the user 10 utters "PCMX" by voice, it is judged to be the same as the external device in the dialog box display frame line 303, and the desktop PC 301 is selected.

[0113] The above ends the external device selection auxiliary process (S400) (S411). Note that the connection process after selecting the external device is based on a general Bluetooth communication connection process, and therefore the description thereof is omitted.

[0114] As described above, according to this embodiment, the relative distance to the external device is calculated using the UWB communication function, and the positions of multiple external devices with Bluetooth communication functions are grasped. In addition, the external device identification information is overlapped and displayed in the external device image extracted from the photographic image, and more specifically, the device name of the external device is overlapped and displayed at the position of the external device. As a result, it is possible to easily determine the appearance and identification information of the external device to be connected from multiple external devices with Bluetooth communication functions based on the grasped positions of the external devices, and make them consistent. As a result, the user can intuitively identify the external devices that can be connected. In addition, even in the case where there are multiple external devices, the external device to be connected can be simply and accurately selected. Moreover, even in the case where multiple external devices have the same device name, as long as the relative positions of the external devices are different, the external device to be connected can be determined.

[0115] Example 2

[0116] In the first embodiment, the device names of the external devices that can be connected are overlapped and displayed in the dialog box display frame line 303, but in this embodiment, a different method of overlapping display is described. In addition, the basic hardware and software structure of this embodiment is the same as that of the first embodiment. Therefore, in this embodiment, the differences from the first embodiment are mainly described, and the common parts are omitted.

[0117] Fig.14A is a schematic diagram showing an example of displaying the external device name in this embodiment, which is an extracted Fig.13 A schematic diagram of a portion of the image shown. Fig.14A In, with Fig.13 The same structures are marked with the same symbols and their descriptions are omitted. Fig.14A In the example, after arranging the device name of the desktop PC 301 as a connectable external device inside the dialog box display frame 303, the outer shape of the desktop PC 301 is further extracted, and the desktop PC 301 is surrounded by a frame 324 so that the extracted external device can be identified and highlighted. The displayed frame 324 can also be further highlighted by a colored frame, a blinking frame, etc. In addition, although not shown in the figure, the entire external device or its outline can also be emphasized, such as by blinking or coloring.

[0118] In addition, sometimes it is not possible to clearly determine the relative position of the external device. Specifically, the intersection of the straight-line distance circle with respect to the UWB antenna is Fig.12 In this case, the center of the intersection of the three points of the straight-line distance circle relative to the UWB antenna can be used as the relative position of the external device. However, in this case, the accuracy of the relative position of the external device is low, so it can also be changed to a display indicating low position accuracy. For example, Fig. 14B As shown, it can be set as a cloud-shaped dialog box frame line 333. Of course, it can also be realized by other identification displays. In addition, the display of the frame line 324 can also be changed according to the extraction accuracy of the appearance of the external device mentioned above.

[0119] In addition, consider the case where there are multiple external devices, and the relative positions and directions of these external devices are consistent. In this case, the external device close to the HMD 1 is displayed in the front, and the external device is displayed in the back. Fig. 14C As shown, the dialog display frame line 306 of the external device at a far distance may be arranged behind the dialog display frame line 303 of the external device at a close distance.

[0120] In addition, as different display methods, such as Fig.14D As shown, it is also possible to set a dialog box 343 to indicate the presence of multiple external devices. Fig.14D In the figure, it is indicated that there are multiple external devices in the double-dotted line dialog box frame line 343, indicating that there are external devices that are close in the first shaded area 344, and there are external devices that are far away in the second shaded area 345. In this display method, when the second shaded area 345 is selected, the device name can be confirmed. Of course, it can also be implemented by other identification displays. Thus, it can be set to a display that can distinguish multiple external devices.

[0121] Thus, according to this embodiment, by studying the emphasized display of connectable external devices, the display indicating low position accuracy, and the display method when there are multiple external devices, the user can more intuitively identify the connectable external devices and can more easily and accurately select the external device they want to connect.

[0122] Example 3

[0123] In the first embodiment, it is assumed that the image of the external device existing in the visual field of the user 10 wearing the HMD 1 can be extracted, but in this embodiment, the response to the case where the image of the external device cannot be extracted is described. In addition, the basic hardware and software structure of this embodiment is the same as that of the first embodiment. Therefore, in this embodiment, the differences from the first embodiment are mainly described, and the common parts are omitted.

[0124] As a situation where the image of the external device cannot be extracted, for example, a situation where the external device is located inside a certain object, a situation where the external device is blocked by other objects, and a situation where the external device cannot be visually confirmed is considered. Fig.15A3 is a schematic diagram showing a state where a smartphone 351 capable of UWB connection exists in a corrugated cardboard box 35 of this embodiment. UWB uses a very wide frequency band and is therefore not easily affected by obstacles. Therefore, a smartphone 351 capable of UWB connection existing in a corrugated cardboard box 35 can be connected to UWB from outside the corrugated cardboard box 35.

[0125] The external device selection auxiliary processing in this embodiment is the same as that in Embodiment 1. Figure 5 However, the image extraction judgment of the external device in S406 and the matching judgment processing of the extracted image and the position in S407 are omitted. By omitting S406 and S407, the relative position of the external device detected in UWB communication is overlapped and displayed with the name of the external device.

[0126] For example, in Fig.15A In the embodiment, the smartphone 351 cannot be directly visually confirmed, but the relative position of the smartphone 351 is indicated by a dotted frame dialog box display 353 in which the device name is superimposed. Therefore, the user 10 can recognize that the smartphone 351 exists in the corrugated box 35. In addition, in the present embodiment, a solid frame dialog box display is set in the case of an external device that can be directly viewed, and a dotted frame dialog box display is set in the case of an external device that cannot be directly viewed, but it is not limited to this display method. In addition, the user 10 can also set whether to omit the processing of S406 and S407. Alternatively, it is also possible to detect the presence of other objects on the front side of the relative position of the detected external device by a distance sensor 55, and only omit the processing of S406 and S407 when other objects are reliably present. For example, if the distance to the smartphone 351 is detected to be 2m, and the distance to the surface of the corrugated box 35 is detected to be 1.5m, it is determined that the smartphone 351 is blocked by the corrugated box 35, and the processing is performed in a manner that omits S406 and S407.

[0127] As described above, in this embodiment, even when the external device is inside another object or is blocked by another object, the device name of the external device is displayed in an overlapping manner. Therefore, the correspondence between the invisible external device and its device name is visually consistent, and the external device that can be connected can be intuitively selected.

[0128] As another example in which the image of the external device cannot be extracted, consider a case in which the external device exists outside the field of view of the user 10. Hereinafter, the case in which the external device exists outside the field of view will be described.

[0129] The external device selection auxiliary processing in the case where there is an external device outside the field of view is the same as that in Example 1 Figure 5However, if the external device position is determined to be outside the visual range in S403, the external device name is superimposed in S408. Fig. 15B 2 is a schematic diagram showing an example of a dialog box display when the relative position of the external device detected in this embodiment is outside the field of view of the HMD 1. Fig. 15B In the example, the name of the external device, the distance to the external device, and the direction of the external device are indicated inside the dialog display 366. The direction of the external device may be indicated by, for example, the front of the HMD 1 as 0 degrees. In addition, when the dialog direction of the dialog display 366 is matched with the direction of the external device, a more intuitive display is achieved. In addition, the user 10 may be able to set whether the processing of S408 must be performed.

[0130] Even when the relative position of the external device is not within the front field of view of the HMD 1, the user 10 can know the relative position and direction of the connectable external device through the dialog display 366. If the direction toward the external device is known, the user 10 wearing the HMD 1 can configure the relative position of the external device within the front field of view of the HMD 1 by changing the orientation of the face to the direction of the external device.

[0131] As described above, according to the present embodiment, even when an image of an external device cannot be extracted, the relative position of the external device can be grasped, and a connectable external device can be intuitively selected.

[0132] Example 4

[0133] In the first embodiment, it is assumed that the relative value of the external device can be detected, but in this embodiment, the response to the case where there is an external device whose relative value cannot be detected is described. In addition, the basic hardware and software structure of this embodiment is the same as that of the first embodiment. Therefore, in this embodiment, the differences from the first embodiment are mainly described, and the common parts are omitted.

[0134] Fig.16 : is a schematic diagram showing a display example when there is an external device whose relative position cannot be detected in this embodiment. Fig.16 In, with Fig.13 The same structures are marked with the same symbols and their descriptions are omitted. Fig.16In the embodiment, the desktop PC 301 can perform position detection using UWB, and through the processing shown in Example 1, the device name is displayed in a overlapping manner in the dialog display frame 303 to match the position of the desktop PC 301. On the other hand, the smartphone outside the display area or outside the field of view of the HMD 302 and the display screen 75 is an external device that is not equipped with UWB. In this case, the HMD 302 and the smartphone outside the field of view cannot perform position detection, so it is impossible to perform overlapping display matching their positions. However, considering the user 10's requirement to connect to these external devices. Therefore, the external devices that cannot perform position detection can also be displayed in a list as 373 and 374. In addition, the position of the external device that cannot perform position detection is unclear, so instead of displaying the prompt position as in the dialog display frame 303, the shape is given a difference in the form of a simple rounded quadrilateral as in 373 and 374 to change the display. The user can judge whether the relative position of the external device can be detected based on whether the display of the external device name is a dialog display or a simple rounded quadrilateral.

[0135] In addition, a conventional connection context menu for selecting a desired external device from a list of external devices can be prepared in HMD 1. In this case, when an external device capable of position detection is selected from the list, the conventional context menu screen can be automatically switched to the superimposed screen display of this embodiment.

[0136] As described above, according to this embodiment, even when there is an external device that cannot detect a position, the user can select the external device to be connected using a unified user interface. In addition, it is of course possible to implement the above embodiment in combination with this embodiment.

[0137] Example 5

[0138] In the first to fourth embodiments, it is assumed that the user selects the external device to be connected in the HMD, but in this embodiment, the case where the HMD is selected and connected from the external device is described. In addition, the basic hardware and software structure of this embodiment is the same as that of the first embodiment. Therefore, in this embodiment, the differences from the first embodiment are mainly described, and the common parts are omitted.

[0139] Fig.17 381 is a schematic diagram showing a situation in which a person 38 wearing an HMD 381 is present in the field of view of the HMD 1 in this embodiment. The device name of the HMD 381 is displayed as a dialog box 383 superimposed on the display screen 75. In this embodiment, it is assumed that the person 38 wearing the HMD 381 selects the HMD 1 worn by the user 10 before the user 10 selects an external device.

[0140] In this embodiment, when HMD 1 detects a connection, the device name and information indicating that a connection request has been received are displayed, and the external device (HMD 381) that is the connection source can be identified. For this purpose, the dialog display 383 is set to a filled display, and the character color of the device name is changed. In addition, by also adding the display of "connection request", the user 10 can easily recognize that HMD 381 is the connection source external device. Of course, other display methods such as flashing the dialog display 383 can also be used.

[0141] As described above, according to the present embodiment, the correspondence between the connection source external device and its device name is visually consistent, and the connection source external device can be intuitively identified.

[0142] Example 6

[0143] In the first to fifth embodiments, HMD is used as an example of a portable information terminal, but in this embodiment, a smartphone is used as an example of a portable information terminal. In addition, the basic hardware and software structure of this embodiment is the same as that of the first embodiment. Therefore, in this embodiment, the differences from the first embodiment are mainly described, and the common parts are omitted.

[0144] Fig.18 This is a schematic diagram for explaining the overall outline of the system in this embodiment. Fig.18 The following situation is expressed in Figure 1 In the HMD 1 , the user 10 holds the smartphone 18 and searches for an external device that can communicate with the smartphone 18 through Bluetooth.

[0145] Fig.19 1 is an external view showing an example of a smart phone 18 as a portable information terminal. Fig.19 In the figure, the upper figure shows the front side, and the lower figure shows the back side. The smartphone 18 has a display screen 181 composed of a touch panel and a front camera 182 on the front side, and a rear camera 184 on the back side. In addition, in the smartphone 18 of this embodiment, as in the HMD 1 in the above embodiment, a right UWB antenna 644 is arranged at the right end of the front side, a left UWB antenna 645 is arranged at the left end of the front side, and a central UWB antenna 646 is arranged at the upper center of the front side.

[0146] The user 10 points the smartphone 18 toward the external device and uses the rear camera 184 to shoot. The captured image is displayed on the display screen 181 of the smartphone 18, and the name of the external device is displayed in a dialog box in a superimposed manner as in the above-mentioned embodiment. In the embodiment of the HMD 1, the feature is that the image is superimposed on the real space image visible through the transmission, but in this embodiment, the image is superimposed on the photographic image. However, the operation and processing algorithm after the superimposed display are the same as those in the above-mentioned embodiment, and the effects in the above-mentioned embodiment can be directly applied.

[0147] As described above, according to the present embodiment, the external device selection assistance process can also be performed in a smartphone.

[0148] Example 7

[0149] In the above-mentioned embodiment, UWB is used to detect the direction and distance of the external device, but in this embodiment, the case where the portable information terminal uses Bluetooth to detect the distance and direction of the external device is described. In addition, the basic hardware and software structure of this embodiment is the same as that of embodiment 1. Therefore, in this embodiment, the differences from embodiment 1 are mainly described, and the common parts are omitted.

[0150] First, regarding the distance detection of the external device, the signal attenuation is measured using the received signal strength (RSSI) value in the Bluetooth communication unit 63, thereby being able to estimate the distance to some extent. As an example, if a value (TxPower) representing the output strength of Bluetooth is obtained from the external device, the distance D to the external device can be calculated by the following formula (4).

[0151] D=10^((TxPower-RSSI) / 20)…(4)

[0152] Here, the units of TxPower and RSSI are dBm, and the calculation formulas are based on the assumption that the portable information terminal and the external device exist in a free space, that is, an ideal space without obstacles.

[0153] The direction detection of the external device can use the Bluetooth angle of arrival (AoA) and angle of departure (AoD) information. The AoA-based method is a method in which a transmitting device with a single antenna sends a direction detection signal, and a receiving device with multiple antennas uses the phase difference between antennas to detect the direction of the transmitter. It uses the change in the phase of the received signal of each receiving antenna due to the difference in the distance between the transmitting and receiving antennas. The AoD-based method is a method in which a transmitter with multiple antennas sends a direction detection signal, obtains the phase information of the received signal from the IQ signal received by the antenna of the receiving device, and detects the direction. In the AoD-based method, the antenna of the receiving device can be one. In this embodiment, any of AoA and AoD can be used.

[0154] The hardware structure diagram of the portable information terminal, i.e., HMD in this embodiment is similar to Figure 3 The same as the embodiment, but uses Bluetooth to detect the distance and direction of the external device, so the UWB communication unit 64 can also be omitted. Figure 4 However, the device position calculation process 26 is replaced by a function of calculating the distance based on the RSSI information of the external Bluetooth device having the Bluetooth communication function by transmitting and receiving between the HMD1 and the external Bluetooth device having the Bluetooth communication function through the Bluetooth connection process 22. Figure 3 In the case of the UWB communication unit 64, the UWB connection process 24 and the UWB device information storage 25 are also omitted.

[0155] In addition, the flowchart of the external device selection auxiliary processing in this embodiment is similar to Figure 5 However, when omitting Figure 3 In the case of the UWB communication unit 64, the UWB connection process S440 is omitted. In addition, the external device distance calculation and storage process is performed in the advertisement / scanning process S420.

[0156] Fig. 20 A flowchart showing the advertisement / scanning process S420 in this embodiment. Fig. 20 and Figure 6 Basically the same, but the UWB-related processing S428 and S429 are omitted. Instead, if the scan response reception judgment S425 is judged to be yes, the external device distance calculation / save S432 is added. In S432, as described above, the distance to the external device is calculated based on the RSSI information. In addition, the direction of the external device is calculated using the method based on AoA or AoD, and the calculated distance and direction information are saved in the external device information table T800.

[0157] Fig.21 It is omitted Figure 3 An example of the external device information table T800 in the case of the UWB communication unit 64. Fig.21 For example, regarding the device with external device number 1, the distance D1 to the external device, the horizontal angle Ph1 of the external device, and the vertical angle Pv1 of the external device calculated in S432 are stored. In this way, the information that can identify the position of the external device is associated with the name of the external device and stored, so that the name of the external device can be displayed in an overlapping manner in the extracted image that matches the position of the external device.

[0158] As described above, according to the present embodiment, the external device selection assistance process can be performed using only Bluetooth as the wireless function.

[0159] The embodiments of the present invention have been described above. The present invention contributes to 9 of the Sustainable Development Goals (SDGs: Sustainable Development Goals) advocated by the United Nations: Building a foundation for industrial and technological innovation by enabling intuitive and reliable selection of external devices.

[0160] In addition, the present invention is not limited to the above-mentioned embodiments, and includes various modified examples. For example, as a portable information terminal, as long as it has the same functions as the embodiments, the present invention can also be applied to portable information terminals other than HMD and smart phones, such as tablet computers, personal computer devices including notebook computers, etc.

[0161] In addition, in the above-described embodiment, as the object of the external device selection auxiliary processing, the external device having the Bluetooth connection function is taken as the object, but it can also be applied to the external device (specifically, the access point) having the wireless LAN connection function. In the external device having the Bluetooth connection function, the device name is used for selection, but in the external device having the wireless LAN connection function, the SSID (Service Set Identifier) ​​which is the service set identifier is used for selection. In addition, the present invention can also be applied to the selection of external devices in other connection systems.

[0162] In addition, a part of the structure of a certain embodiment can be replaced with the structure of other embodiments, and the structure of other embodiments can be added to the structure of a certain embodiment. All of these belong to the scope of the present invention. In addition, the numerical values, messages, etc. appearing in the text and the figure are only examples, and even if different numerical values ​​and messages are used, the effect of the present invention will not be impaired. In addition, the functions of the present invention, etc. described above can also replace the software processing described, for example, by using hardware to implement part or all of them by designing in an integrated circuit. In addition, hardware and software can also be used together. The software can also be in a state pre-stored in the program unit 41 of HMD1, smart phone 18, etc. at the time of product shipment. After the product is shipped, it can also be obtained from various server devices on the Internet. In addition, software provided by a memory card, a CD, etc. can also be obtained.

[0163] Explanation of symbols

[0164] 1: HMD, 2, 902: main control unit, 4, 904: storage unit, 5: sensor unit, 6, 906: communication processing unit, 7: image processing unit, 8: sound processing unit, 9: operation input unit, 16: network server, 18: smart phone, 42: various data units, 51: GPS receiving unit, 63: Bluetooth communication unit, 64, 964: UWB communication unit, 71: camera unit, 72: display unit, 75: display screen, 184: rear camera, 301: desktop PC, 303: display frame line, 641, 644: right UWB antenna, 642, 645: left UWB antenna, 643, 646: center UWB antenna, 711: right camera, 712: left camera.

Claims

1. A portable information terminal capable of wirelessly connecting to an external device, characterized in that: The portable information terminal comprises: A first communication function and a second communication function capable of wirelessly connecting to the external device; A camera for photographing the surroundings of the portable information terminal; an image processing unit, which processes the image captured by the camera; a display, which displays the image processed by the image processing unit; a storage unit storing first identification information of the external device used in the first communication function and second identification information of the external device used in the second communication function; as well as Control Department, The control unit calculates a relative position of the external device with respect to the portable information terminal by using the second communication function. The image processing unit generates a superimposed image in which the first identification information is superimposed on the image at the calculated relative position of the external device, and displays the superimposed image on the display.

2. The portable information terminal according to claim 1, characterized in that: The control unit can acquire the first identification information even when the second communication function is used, or can acquire the second identification information even when the first communication function is used.

3. The portable information terminal according to claim 2, characterized in that: The storage unit stores the first identification information and the second identification information in association with each other for each external device.

4. The portable information terminal according to claim 1, characterized in that: The control unit extracts the external device's outer shape at a relative position to the external device from the image captured by the camera. The image processing unit performs image processing capable of identifying the extracted external device.

5. The portable information terminal according to claim 1, characterized in that: The first communication function is a Bluetooth communication function, the second communication function is a UWB communication function, The first identification information is a device name of the external device.

6. The portable information terminal according to claim 1, characterized in that: The image processing unit changes display of the superimposed image according to the calculated accuracy of the relative position of the external device.

7. The portable information terminal according to claim 4, characterized in that: The image processing unit changes display of the superimposed image according to extraction accuracy of the extracted outer shape of the external device.

8. The portable information terminal according to claim 1, characterized in that: When there are a plurality of calculated relative positions of the external devices within a certain area, the image processing unit changes display of the superimposed image so that the plurality of external devices can be distinguished.

9. The portable information terminal according to claim 1, characterized in that: When the calculated relative position of the external device is outside the display area of ​​the display, the image processing unit changes the display of the superimposed image from that when the calculated relative position of the external device is within the display area.

10. The portable information terminal according to claim 9, characterized in that: The image processing unit generates a superimposed image indicating the first identification information and the calculated relative position direction of the external device.

11. The portable information terminal according to claim 1, characterized in that: When receiving a connection request from the external device, the image processing unit generates a superimposed image in which the first identification information and information indicating that the connection request has been received are superimposed at a relative position of the external device.

12. The portable information terminal according to claim 1, characterized in that: The portable information terminal comprises: a selection unit which selects the external device; The selection unit can process two methods: a first selection method for selecting the overlapped first identification information and a second selection method for selecting by displaying a list of the wirelessly connectable external devices. The control unit switches to the first selection method when the external device selected by the second selection method is capable of relative position calculation.

13. An external device capable of wirelessly connecting to a portable information terminal, characterized in that: The external device has: a first communication function and a second communication function capable of wirelessly connecting to the portable information terminal; and a storage unit storing first identification information used in the first communication function and second identification information used in the second communication function, In addition to sending the first identification information through the first communication function and sending the second identification information through the second communication function, the second identification information is also sent through the first communication function.

14. The external device according to claim 13, characterized in that: The first communication function and the second communication function are respectively a Bluetooth communication function and a UWB communication function, or a UWB communication function and a Bluetooth communication function.

15. An information processing method for a portable information terminal capable of wirelessly connecting to an external device, characterized in that: calculating the relative position of the external device with respect to the portable information terminal by using a second communication function capable of wirelessly connecting to the external device, A superimposed image is generated and displayed in which first identification information of the external device used in a first communication function capable of wirelessly connecting with the external device is superimposed at the calculated relative position of the external device in an image captured around the portable information terminal.

16. An information processing method for an external device capable of wirelessly connecting to a portable information terminal, characterized in that: In addition to transmitting the first identification information used in the first communication function through the first communication function capable of wireless connection with the portable information terminal, In addition to transmitting the second identification information used in the second communication function through the second communication function capable of wirelessly connecting to the portable information terminal, the second identification information is also transmitted through the first communication function.

17. A portable information terminal capable of wirelessly connecting to an external device, characterized in that: The portable information terminal comprises: A Bluetooth communication function capable of wirelessly connecting to the external device; A camera for photographing the surroundings of the portable information terminal; an image processing unit, which processes the image captured by the camera; a display, which displays the image processed by the image processing unit; a storage unit storing a device name of the external device used in the Bluetooth communication function; as well as Control Department, The control unit calculates the relative position of the external device with respect to the portable information terminal by using the Bluetooth communication function. The image processing unit generates a superimposed image in which the device name is superimposed at the calculated relative position of the external device in the image, and displays the superimposed image on the display.

Images