Portable information terminal and display method therefor

Abstract

To provide a portable information terminal and a display method therefor that present the estimated position of a radio wave source in a manner easily understandable by a user when searching for the radio wave source.SOLUTION: Provided is a portable information terminal comprising a wireless reception device for receiving a wireless signal of a radio wave source, a sensor for measuring the movement amount and rotation amount of the terminal, a display device, and a control device. The control device estimates the position of the radio wave source from the reception strength of the radio wave source received by the wireless reception device and the movement amount and rotation amount of the terminal measured by the sensor, and displays a diagrammatic guide display indicating a three-dimensional direction and distance indicating the relative positional relationship of the estimated position of the radio wave source as seen from the position of the portable information terminal, on the display device, to be superimposed on an exterior image.SELECTED DRAWING: Figure 2A

Description

[Technical field]

[0001] The present invention relates to a portable information terminal for locating a radio wave source, which estimates and displays the position of a radio wave source such as an RF tag or a beacon. [Background technology]

[0002] Background art in the technical field of radio wave source position estimation is provided by Patent Document 1. Patent Document 1 discloses a method in which a mobile terminal is moved to different positions to measure the strength of the radio wave received from a transmitter and the position of the mobile terminal itself, which is the receiver, and the position of the transmitter is estimated by integrating the strength of the radio wave and the receiver position information as the measurement results obtained at multiple points, and the position of the estimated radio wave source is displayed by overlaying it on two-dimensional map information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-142180 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, since the position needs to be indicated on a two-dimensional map, first, there is a problem that it cannot be used in an environment where map information cannot be acquired. In addition, in the case of indoors, in an environment where indoor positioning is not possible, the terminal's own position on the map is unknown, and therefore the location of the radio wave source on the map is also unknown. Furthermore, there are cases where an actual object not shown on the map information is present, and the estimated radio wave source position is difficult to understand because it is hidden by the object. Another problem is that information on the height of the radio wave source is missing.

[0005] In view of the above-mentioned problems, the present invention has an object to provide a portable information terminal and a display method thereof that, when searching for a radio wave source, presents an estimated radio wave source position to a user in an easily understandable manner. [Means for solving the problem]

[0006] One example of the present invention is a portable information terminal equipped with a wireless receiving device that receives a wireless signal from a radio wave source, a sensor that measures the amount of movement and rotation of the terminal itself, a display device, and a control device, in which the control device estimates the position of the radio wave source from the reception strength of the radio wave source received by the wireless receiving device and the amount of movement and rotation of the terminal itself measured by the sensor, and is configured to display on the display device a graphical guide display indicating three-dimensional direction and distance showing the relative positional relationship of the estimated position of the radio wave source from the position of the portable information terminal, superimposed on an external world image. Effect of the Invention

[0007] According to the present invention, it is possible to provide a portable information terminal and a display method thereof that, when searching for a radio wave source, presents an estimated radio wave source position to a user in an easily understandable manner. [Brief description of the drawings]

[0008] [Figure 1] 1 is a schematic configuration diagram of a radio wave source location system according to a first embodiment. [Figure 2A] 13 is a display example of an estimated position of a radio wave source when the portable information terminal in the first embodiment is an HMD. [Figure 2B] 2B is a display example of an estimated position of a radio wave source when the position of the radio wave source in FIG. 2A has changed. [Diagram 3] 2 illustrates an example of the external configuration of an HMD as an example of a portable information terminal in the first embodiment. [Figure 4] 2 is a functional block configuration example of a portable information terminal (HMD) in the first embodiment. [Diagram 5] FIG. 2 is a process flow diagram of radio wave source search in the first embodiment. [Figure 6] FIG. 2 is an explanatory diagram of direction estimation of a radio wave source in the first embodiment. [Figure 7] 13 is a modified example of the display of the estimated position of the radio wave source in the first embodiment. [Figure 8] 13 is a display example of an estimated position of a radio wave source in the second embodiment. [Figure 9]FIG. 11 is a process flow diagram of radio wave source search in the second embodiment. [Figure 10] 13 is a display example of an estimated position of a radio wave source in the third embodiment. [Figure 11] 13 is a display example of an estimated position of a radio wave source in the fourth embodiment. [Figure 12] FIG. 13 is a schematic configuration diagram of a radio wave source location system according to a fifth embodiment. [Figure 13] FIG. 13 is a process flow diagram of radio wave source search in the fifth embodiment. [Figure 14A] 23 is a display example of an estimated position of a radio wave source when the mobile information terminal in the sixth embodiment is a smart watch. [Figure 14B] 14B is a display example of an estimated position of the radio wave source in the case where the radio wave source in FIG. 14A is located in a position blocked by a structure. [Figure 15A] 23 is another display example of the estimated position of the radio wave source when the mobile information terminal in the sixth embodiment is a smart watch. [Figure 15B] This is a display example of the estimated position of the radio wave source when the radio wave source is in an upward direction in FIG. 15A. [Figure 16] FIG. 13 is a schematic configuration diagram of a partner terminal search system in a seventh embodiment. [Figure 17A] 13 is a display example of an estimated location of a counterpart terminal in the seventh embodiment. [Figure 17B] 13 is another display example of the estimated location of the counterpart terminal in the seventh embodiment. [Figure 18] FIG. 23 is a process flow diagram of a partner terminal search in the seventh embodiment. [Figure 19A] 23 is a display example of an estimated position of a radio wave source in the eighth embodiment. [Figure 19B] 19B is a display example of the estimated position of the radio wave source when the accuracy of the position estimation of the radio wave source in FIG. 19A is improved. [Figure 20] FIG. 23 is a process flow diagram of radio wave source search in the eighth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. EXAMPLES

[0010] FIG. 1 is a schematic diagram of a radio wave source location system in this embodiment. In FIG. 1, the radio wave source location system has at least one of the following mobile information terminals for searching and estimating the position of a radio wave source 80: mobile information terminal 1A which is an HMD (Head Mount Display), mobile information terminal 1B which is a smartphone, and mobile information terminal 1C which is a smart watch. The following radio wave source search operation of the mobile information terminals may be performed by these mobile information terminals alone or in cooperation with each other. In the following description, mobile information terminals 1A to 1C are collectively referred to as mobile information terminal 1. Also, the mobile information terminal may be simply referred to as a terminal.

[0011] The server 2 performs processing on behalf of each mobile information terminal 1, mediates the transmission and reception of information between each mobile information terminal 1, and provides necessary information via a communication network 9. The server 2 may be, for example, a local server, a cloud server, an edge server, a net service, or the like, and its form is not important.

[0012] 2A is a display example of an estimated position of a radio wave source when the mobile information terminal in this embodiment is an HMD. In FIG. 2A, the outer frame represents the field of view of the user when the HMD is an optical see-through type, and represents the field of view of the display when the HMD is a video see-through type.

[0013] As shown in FIG. 2A, a circle indicated by 81 is a mark position that displays the estimated position of the radio wave source (hereinafter, referred to as the estimated position of the radio wave source 81), and the estimated position of the radio wave source 81 and a guide 82 that is an object that allows the wearer of the HMD to intuitively understand the sense of distance are displayed. For example, as shown in FIG. 2A, the guide 82 may be an object that allows a certain interval to be understood. Also, the estimated position of the radio wave source 81 and the guide 82 are displayed superimposed on an image of the outside world. In this way, the display of the estimated position of the radio wave source in this embodiment displays, together with the estimated position of the radio wave source, a graphical guide display that shows a three-dimensional direction and distance that shows the relative positional relationship of the estimated position of the radio wave source superimposed on the image of the outside world. Note that, in FIG. 2A, the distance to the estimated position of the radio wave source may be displayed as shown at 83. Also, as shown at 84, the interval between points that serves as a guide for the distance may be displayed.

[0014] Fig. 2B is a display example of the estimated position of the radio wave source when the position of the radio wave source has changed in Fig. 2A. As shown in Fig. 2B, by aligning the display position of the tip of the guide 82 with the estimated position 81 of the radio wave source in the field of view, that is, by overlapping them, the location of the estimated position of the radio wave source is emphasized and becomes easier to understand. In addition, since the distance to the target is known, it is easy to recognize the position even if it is hidden by an object.

[0015] Fig. 3 is an example of the external configuration of an HMD as an example of a portable information terminal in this embodiment. In Fig. 3, a portable information terminal 1A, which is an HMD, is provided with a display device including a display surface 11 in a glasses-like housing 10. This display device is, for example, a transmissive display device, and a real image of the outside world is transmitted to the display surface 11, and an image is superimposed and displayed on the real image. A control device, a camera 12, a distance measuring sensor 13, other sensor units 14, etc. are mounted on the housing 10.

[0016] The camera 12 has, for example, two cameras arranged on the left and right sides of the housing 10, and captures an image by capturing an area including the front of the HMD. The distance measurement sensor 13 is a sensor that measures the distance between the HMD and an object in the outside world. The distance measurement sensor 13 may use a TOF (Time Of Flight) type sensor, a stereo camera, or other types. The sensor unit 14 includes a group of sensors for detecting the position and orientation state of the HMD. On the left and right sides of the housing 10, there are provided an audio input device 18 including a microphone, an audio output device 19 including a speaker and an earphone terminal, and the like.

[0017] The mobile information terminal 1A, which is an HMD, may be provided with an operation device 20 such as a remote controller. In this case, the HMD performs, for example, short-distance wireless communication with the operation device 20. A user can input instructions related to the functions of the HMD and move a cursor on the display surface 11 by operating the operation device 20 with his or her hand. The HMD may communicate with an external smartphone, PC, or the like to cooperate with the HMD. For example, the HMD may receive AR (Augmented Reality) image data from an application on the smartphone.

[0018] The mobile information terminal 1A, which is an HMD, may display a virtual image such as an AR image on the display surface 11. For example, the mobile information terminal 1A, which is an HMD, generates a virtual image for guiding the user and displays it on the display surface 11.

[0019] Fig. 4 is a functional block diagram of a mobile information terminal 1A, which is the HMD in Fig. 3. Basically, the other mobile information terminals 1 have a similar configuration. The mobile information terminal 1 includes a processor 101, a memory 102, a camera 12, a distance measuring sensor 13, a sensor unit 14, a display device 103, a communication device 104, an audio input device 18 including a microphone, an audio output device 19 including a speaker, etc., a wireless transmitting device 105, a wireless receiving device 106, an operation input unit 107, and a battery 108. These elements are connected to each other via a bus or the like.

[0020] The processor 101 is composed of a CPU, a ROM, a RAM, etc., and constitutes a control device for the HMD. The processor 101 executes processes according to a control program 31 and an application program 32 in the memory 102, thereby realizing functions of an OS, middleware, application, etc., and other functions. The memory 102 is composed of a non-volatile storage device, etc., and stores various data and information handled by the processor 101, etc. The memory 102 also stores images and detection information acquired by the camera 12, etc., as temporary information.

[0021] The camera 12 converts light incident from a lens into an electrical signal using an imaging element to obtain an image. When the distance measurement sensor 13 uses, for example, a TOF (Time Of Flight) sensor, it calculates the distance to an object from the time it takes for light emitted to the outside world to hit the object and return. The sensor unit 14 includes, for example, an acceleration sensor 141, a gyro sensor (angular velocity sensor) 142, a geomagnetic sensor 143, and a GPS receiver 144. The sensor unit 14 detects the position, orientation, movement, and other conditions of the HMD using detection information from these sensors. The HMD is not limited to these, and may also include an illuminance sensor, a proximity sensor, an air pressure sensor, and the like.

[0022] The display device 103 includes a display drive circuit and a display surface 11, and displays a virtual image or the like on the display surface 11 based on image data of the display information 34. Note that the display device 103 is not limited to a transmissive display device, and may be a non-transmissive display device or the like.

[0023] The communication device 104 includes a communication processing circuit, an antenna, and the like corresponding to various predetermined communication interfaces. Examples of the communication interface include a mobile network, Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared, and the like. The communication device 104 performs wireless communication processing between other mobile information terminals 1 and access points. The communication device 104 also performs short-range communication processing with a controller.

[0024] The wireless transmitting device 105 transmits a wireless signal of a radio wave source accompanied by an identification signal used for position estimation. Here, the wireless signal accompanied by identification information is, for example, a signal such as Bluetooth. The wireless receiving device 106 receives the wireless signal of the radio wave source accompanied by the identification signal and measures the reception strength. Note that information communication may be performed using the wireless transmitting device 105 and the wireless receiving device 106. In addition, in the case of a mobile information terminal 1 that only transmits or only receives wireless signals, it is sufficient to have the necessary devices.

[0025] The audio input device 18 converts input audio from a microphone into audio data. The audio output device 19 outputs audio from a speaker or the like based on the audio data. The audio input device may have a voice recognition function. The audio output device may have a voice synthesis function. The operation input unit 107 is a part that accepts operation inputs to the HMD, such as power on / off and volume adjustment, and is composed of hardware buttons, touch sensors, etc. The battery 108 supplies power to each part.

[0026] The control device by the processor 101 has, as an example of a configuration of functional blocks realized by processing, a communication control unit 101A, a display control unit 101B, a data processing unit 101C, and a data acquisition unit 101D.

[0027] The memory 102 stores a control program 31, an application program 32, setting information 33, display information 34, position estimation information 35, etc. The control program 31 is a program for realizing an estimation of the relative positional relationship between the mobile information terminals 1. The application program 32 is a program for realizing a guidance function for the user. The setting information 33 includes system setting information and user setting information related to each function. The display information 34 includes image data and position coordinate information for displaying a virtual image on the display surface 11. The position estimation information 35 is information related to the moving distance of the mobile information terminal 1, the orientation of the mobile information terminal 1 with respect to the outside world, and the reception strength of a wireless signal for performing position estimation.

[0028] The communication control unit 101A controls communication processing using the communication device 104, such as when communicating with another mobile information terminal 1. The display control unit 101B controls the display of a virtual image or the like on the display surface 11 of the display device 103 using the display information 34.

[0029] The data processing unit 101C reads and writes the position estimation information 35, and performs operations such as estimating the relative positional relationship between the terminal itself and the other terminal.

[0030] The data acquisition unit 101D acquires the strength of a wireless signal from the wireless receiving device 106, and acquires each detection data from various sensors such as the camera 12, the distance measuring sensor 13, and the sensor unit 14. The data acquisition unit 101D estimates its own position and measures the travel distance from the detection data of the various sensors.

[0031] FIG. 5 is a process flow diagram of radio wave source search in this embodiment. In FIG. 5, the mobile information terminal 1 first receives a search start instruction in step S1, and then receives a radio signal of the radio wave source by the wireless receiving device 106 in step S2. Since this is the basis for grasping the relative positional relationship between the radio wave source in the outside world and the mobile information terminal 1, the mobile information terminal 1 needs to constantly measure the positional change and the orientation change of the mobile information terminal 1 in the outside world. Therefore, the mobile information terminal 1 also measures the amount of movement and the amount of rotation, which is the change in orientation of the mobile information terminal 1, from the detection data of the sensor of the sensor unit 14 in FIG. 4 in step S2. Then, in step S3, the processor 101 estimates the position of the radio wave source, and in step S4, displays the estimated position of the radio wave source and the guide. Then, until an instruction to end the search is received, the processes from steps S2 to S4 are continued, and the display of the estimated position of the radio wave source and the display of the guide are continued to be updated. In this way, while moving in search of the radio wave source, the position of the radio wave source is estimated at any time, and the display of the estimated position of the radio wave source and the display of the guide are continued to be updated.

[0032] When estimating the radio wave source position by integrating measurements at multiple points, it is necessary to know the relationship between the current and past positions and orientations of the terminal itself. Since the mobile information terminal 1 is always aware of the changes in the amount of movement and orientation of the terminal itself relative to the outside world from the measurements of the amount of movement and rotation of the terminal itself in step S2, it can also know the relationship between the current and past positions and orientations of the terminal itself.

[0033] An example of a method for estimating the location of a radio source is as follows.

[0034] The received strength of radio waves is inversely proportional to the square of the distance when the reflected waves are sufficiently small. Using a function that represents this characteristic, the location of the radio source can be estimated by fitting data from multiple points using the least squares method. In other words, the received strength P is calculated as follows:

[0035] P = k / {(X-X0) 2 +(Y-Y0) 2 +(Z-Z0) 2} Here, k is a coefficient that depends on the strength of the radio wave source, (X0, Y0, Z0) is the radio wave source position, and (X, Y, Z) is the receiving position.

[0036] Therefore, the reception strength P is measured at multiple reception positions, and k and (X0, Y0, Z0), i.e., the radio wave source position, are found by parameter fitting. When the reflected wave is strong, an error occurs, but as the radio wave source is approached, the reflected wave becomes relatively weaker, improving the estimation accuracy. The fitting function is not limited to the above, and it is also possible to incorporate higher-order terms taking into account distortion caused by reflected waves. Note that if the calculation load for position estimation is large, the calculations may be performed on a server or other device rather than on the mobile information terminal.

[0037] Further, another example of a method for estimating the position of a radio wave source is as follows.

[0038] Using UWB (Ultra Wide Band) technology, it is possible to estimate the location of a radio wave source from measurements at one location. Figure 6 is an explanatory diagram of estimating the direction of a radio wave source. In Figure 6, the direction θ of a radio wave source 80 relative to an antenna array plane 70 can be determined using the AOA (Angle of Arrival) method. In addition, the location of the radio wave source can be determined from distance information d from the radio wave source.

[0039] The distance to the radio source can be determined in two ways, for example: (1) If the terminal is time-synchronized, measuring the time of arrival of radio waves. (2) Measure the time it takes for the radio source to respond to a trigger sent from the terminal. In the above method, the average value of the estimated radio wave source positions at a plurality of locations may be used to reduce errors.

[0040] Fig. 7 shows a modified example of the display of the estimated radio wave source position in this embodiment. In Fig. 7, if the guide 82 is displayed so as to avoid the line from the HMD wearer to the estimated radio wave source position 81, the path to the estimated radio wave source position 81 can be clearly seen, so the tip position of the guide 82 may be displayed to the side, away from the position of the mark that displays the estimated radio wave source position. In other words, the tip of the guide 82 may not overlap the display of the estimated radio wave source position.

[0041] In this way, in this embodiment, a graphical guide display showing the three-dimensional relative positional relationship of the estimated position of the radio wave source from the portable information terminal carried by the user, including the HMD, i.e., the three-dimensional direction and distance from the portable information terminal position, is superimposed on the outside world image. This allows the user to intuitively understand the positional relationship between the actual visible outside world and the radio wave source, making it easy to recognize the position of the radio wave source.

[0042] As described above, according to the present embodiment, when searching for a radio wave source, it is possible to provide a portable information terminal and a display method thereof that present an estimated radio wave source position to a user in an easily understandable manner. EXAMPLES

[0043] FIG. 8 is a display example of the estimated position of the radio wave source when the portable information terminal in this embodiment is an HMD.

[0044] As shown in Fig. 8, when the estimated position of the radio wave source is in a position where it cannot be directly confirmed from the outside, such as being hidden behind an object or being inside a bag, the part of the guide 82 that is blocked by the external object is displayed in a format different from that of the other parts. That is, in Fig. 8, the blocked part is displayed with a dotted line and the unblocked part is displayed with a solid line so that the display of the guide 82 reflects the blocking relationship with the actual object in the outside world. Note that other display examples include displaying in different shades or colors. Similarly, the estimated position of the radio wave source 81 may be displayed in a different format, such as a dotted line. This has the effect of making it easier to understand the positional relationship between the estimated position of the radio wave source and the object in the outside world.

[0045] Fig. 9 is a process flow diagram of radio wave source search in this embodiment. In Fig. 9, the same processes as those in Fig. 5 are given the same reference numerals, and their explanations are omitted. Fig. 9 differs from Fig. 5 in that step S2 is replaced with step S10. That is, in step S10, in order to reflect the shielding relationship in the display of the guide 82, measurement of the distance to an external object is further added to the process of step S2. EXAMPLES

[0046] In this embodiment, when there is anisotropy in the error of the estimated position of the radio wave source, the estimated position of the radio wave source is displayed in a three-dimensional format that reflects the anisotropy.

[0047] FIG. 10 is a display example of the estimated position of the radio wave source in this embodiment.

[0048] In the case of the method of fitting data from multiple points using the least squares method, which is one of the methods of estimating the position of the radio wave source described above, if measurements are taken while moving on a horizontal plane, the vertical spread of the measured positions disappears, and the vertical position estimation error may become large. In such a case, as shown in Figure 10, the estimated position of the radio wave source is displayed with a mark 85 that extends in the vertical direction, which is the direction in which the error is large. In other words, the error range of the estimated position of the radio wave source is displayed including the difference due to the three-dimensional direction.

[0049] This provides a rough guide to the error and also serves as a reference when increasing the number of measurement positions. Note that guidance 86 for specific additional measurement points to reduce the error in the estimated position of the radio wave source may be displayed. EXAMPLES

[0050] In this embodiment, an example will be described in which, when there are multiple target radio wave sources, such as in a shopping center where multiple stores are located using radio beacons installed in the stores, the estimated locations of the multiple radio wave sources are displayed together with a guide.

[0051] Fig. 11 is a display example of the estimated position of a radio wave source in this embodiment. In Fig. 11, a plurality of radio wave sources are distinguished and measured based on the ID information of the radio wave source contained in the radio wave signal, and for example, a first estimated position 87 of the radio wave source and a second estimated position 88 of the radio wave source are displayed.

[0052] Various display methods can be used. That is, the display mode of the estimated position and guide may be changed depending on the distance difference between the estimated position of the radio wave source and the near side. In FIG. 11, the more distant the radio wave source, the smaller the size of the mark displaying the estimated position of the radio wave source (the farther second estimated position of the radio wave source 88 is smaller than the closer first estimated position of the radio wave source 87), and the line of the guide 82 is also made thinner. Note that the color may be changed depending on the distance. Furthermore, only the guide to the near radio wave source may be displayed so that the outside world appears clear.

[0053] Furthermore, when map information is available, the mobile information terminal 1 may display, as an AR object, a guide 89 of the direction of travel to reach the radio wave source. This allows the user to grasp the sense of distance to the destination point by displaying the guide showing the distance to the radio wave source and the direction of travel, which increases the user's sense of security. This guidance is also effective when there is only one radio wave source. When there are multiple radio wave sources, the user may be guided by a route to go around from the nearest radio wave source. The map information may be recorded in the mobile information terminal, or may be obtained from the server 2.

[0054] According to this embodiment, since a plurality of estimated positions of radio wave sources can be confirmed at once, convenience is improved. EXAMPLES

[0055] In this embodiment, an example will be described in which the position of a radio wave source is estimated by a smartphone, and the estimated position of the radio wave source and a guide are displayed on an HMD.

[0056] Fig. 12 is a schematic diagram of the radio wave source search system in this embodiment. In Fig. 12, the same components as those in Fig. 1 and Fig. 3 are given the same reference numerals, and their explanations are omitted. Fig. 12 differs from Fig. 1 and Fig. 3 in that HMD 1A is equipped with four radio wave sources (30a to 30d).

[0057] As shown in Fig. 12, in this embodiment, in order to measure the positional relationship between the smartphone 1B and the HMD 1A, three or more UWB radio wave sources are mounted on the HMD 1A, and the positions of the radio wave sources are measured from the smartphone. Fig. 12 shows an example in which four radio wave sources are mounted (30a to 30d). Based on the information on the positional relationship between the smartphone 1B and the HMD 1A, the target radio wave source position measured by the smartphone 1B is converted to a position in the coordinate system of the HMD 1A.

[0058] Fig. 13 is a process flow diagram of radio wave source search in this embodiment. In Fig. 13, mobile information terminal 1A is an HMD, and mobile information terminal 1B is a smartphone. First, in steps S21A and S21B, a user instruction to start a search may be given to either mobile information terminal, and communication between mobile information terminals 1A and 1B triggers the instruction to start a search to either mobile information terminal, which starts the process of radio wave source search in each of mobile information terminals 1A and 1B.

[0059] In step S22B, mobile information terminal 1B receives a wireless signal and measures the amount of movement and rotation of its own terminal in the same manner as in step S2 described in Fig. 5. Then, in step S23B, it estimates the position of the radio wave source in the same manner as in step S3 described in Fig. 5.

[0060] Next, in step S24B, the positional relationship between the mobile information terminals 1A and 1B is measured. Note that this process is also performed in step S24A of the mobile information terminal 1A. Then, in step S25B, information on the estimated position of the radio wave source estimated in step S23B is transmitted and provided to the mobile information terminal 1A.

[0061] Then, in step S27B, it is determined whether an instruction to end the search has been received, and the processing of steps S22B to S25B is continued until the instruction to end the search is received, and the estimated position of the radio wave source is continued to be updated. Note that the user instruction to end the search may be given to either mobile information terminal, and the processing of the radio wave source search of each mobile information terminal 1A, 1B is terminated by the instruction to end the search given to either mobile information terminal through communication between the mobile information terminals 1A, 1B.

[0062] In step S22A, mobile information terminal 1A measures the distance to the object in the outside world in order to display the blocking relationship. Note that if the blocking relationship is not reflected in the guide display, the measurement does not have to be performed.

[0063] Next, in step S24A, the positional relationship between the portable information terminals 1A and 1B is measured, and in step S25A, information on the estimated position of the radio wave source transmitted from the portable information terminal 1B is received.

[0064] Then, in step S26A, based on information about the positional relationship between mobile information terminals 1A and 1B, the estimated position of the radio wave source transmitted from mobile information terminal 1B is converted to an estimated position of the radio wave source in the coordinate system of mobile information terminal 1A, and the estimated position of the radio wave source and a guide are displayed in the same manner as in step S4 described in Figure 5.

[0065] Thereafter, in step S27A, it is determined whether an instruction to end the search has been received, and until an instruction to end the search is received, the processes of steps S22A to S26A are continued, and the estimated position of the radio wave source and the display of the guide are continually updated.

[0066] Although the cooperation relationship between the mobile information terminals is not shown in Fig. 1, the processing may be distributed among a plurality of mobile information terminals as in this embodiment. In addition, if the relative positional relationship between the mobile information terminals is known, the radio wave reception processing may be performed cooperatively.

[0067] As described above, in this embodiment, the radio wave source position can be intuitively grasped by displaying the image on the HMD. Also, the processing load on the HMD can be reduced by performing calculations for position estimation on the smartphone. EXAMPLES

[0068] In this embodiment, a display example in which the display area for the estimated radio wave source position and the guide is small will be described.

[0069] FIG. 14A is a display example of the estimated radio wave source position when the mobile information terminal in this embodiment is a smart watch. In FIG. 14A, in the case of a mobile information terminal with a small display area such as the smart watch 1C, the tip of the guide display is not aligned with the estimated radio wave source position as shown in FIG. 2B so that the guide display does not become small and difficult to see. That is, in FIG. 14A, the smart watch 1C does not align the tip of the guide 91 with the estimated radio wave source position, and displays the guide 91 with an upward inclination in the depth direction, a downward in the front direction, and a left-right inclination in the left-right direction with respect to the estimated radio wave source position 81, based on the position of the smart watch 1C, and displays an intuitive sense of the way to the estimated radio wave source position in the shape of a guide (here, an arrow). In other words, the direction of the arrow indicates the direction of the radio wave source, and the length indicates the distance to the radio wave source. Note that, as shown in FIG. 14A, the distance and angle to the estimated radio wave source position may be displayed as shown in 92, or a scale indicating the distance of the length of the guide 91 may be displayed as shown in 93.

[0070] Fig. 14B is a display example of the estimated position of the radio wave source when the position of the radio wave source in Fig. 14A has changed and the radio wave source is now in a position blocked by a structure. As shown in Fig. 14B, when the radio wave source is in a position blocked by a structure, the tip of the arrow of the guide 91 is shaded and the display form of the tip of the arrow is changed.

[0071] The image of the outside world on which the guide 91 is superimposed is an image from the user's viewpoint and includes a radio wave source. This image does not have to be captured at the time the guide is displayed. Furthermore, it may be captured by another terminal.

[0072] FIG. 15A is another display example of the estimated position of the radio wave source when the mobile information terminal in this embodiment is a smart watch. In FIG. 15A, the same components as those in FIG. 14A are assigned the same reference numerals, and the description thereof will be omitted. In FIG. 15A, the difference from FIG. 14A is that the smart watch 1C indicates whether the direction of the radio wave source is downward or upward with respect to the horizontal plane or display surface of the smart watch 1C by the shape of the arrow of the guide 91, that is, whether it is tapered or thickened. That is, in FIG. 15A, the arrow is tapered, indicating that the radio wave source is in the downward direction. FIG. 15B is a display example indicating that the radio wave source is in the upward direction.

[0073] Instead of using a tapered or wide arrow shape, any other distinguishable display such as a different color may be used. Also, the display mode may be changed, such as changing the rate of change in the widening of the arrow depending on the inclination. This has the effect of making it possible to estimate the height of the radio wave source even on a small display screen.

[0074] Also, if the inclination relative to the horizontal plane is displayed, the height can be found by changing the height of the smartwatch 1C to find the height that indicates the horizontal. Also, if the inclination relative to the display surface is displayed, the position of the radio wave source can be intuitively found by changing the inclination of the display surface to find the angle at which the inclination of the arrow relative to the display surface is 0. EXAMPLES

[0075] In this embodiment, an example will be described in which, when both people carry portable information terminals with a radio wave source search function, each person can estimate the terminal position of the other person.

[0076] Fig. 16 is a schematic diagram of a radio wave source location system in this embodiment. In Fig. 16, the same components as in Fig. 1 are given the same reference numerals, and their explanations are omitted. In Fig. 16, the difference from Fig. 1 is that the mobile information terminals 1A and 1D, which are HMDs, have a radio wave source location function, and each estimates the terminal position of the other party.

[0077] Fig. 17A is a display example of the estimated location of the remote terminal in this embodiment. In Fig. 17A, the remote terminal estimated location 94 is displayed, and a display 95 is displayed indicating whether the remote terminal has captured the terminal itself, i.e., whether the location has been estimated with sufficient accuracy. This allows the status of the remote terminal to be grasped.

[0078] Fig. 17B is another display example of the estimated location of the remote terminal in this embodiment. In Fig. 17B, the remote terminal estimated location 94 is displayed, and the remote terminal's movement history 96 is also displayed. The movement history can be known only from the measurement results of the own terminal, but if movement information can be obtained from the remote terminal, the change in the estimated location can be known precisely.

[0079] Fig. 18 is a process flow diagram of the other terminal search in this embodiment. In Fig. 18, first, in steps S31A and S31D, a user instruction to start a search may be given to either mobile information terminal, and the search start instruction to either mobile information terminal is used as a trigger to start the other terminal search process of each mobile information terminal 1A, 1D by communication between the mobile information terminals 1A and 1D.

[0080] The subsequent processes for searching for a partner terminal in each of the portable information terminals 1A and 1D are similar, so the process in the portable information terminal 1A will be representatively explained.

[0081] In step S32A, mobile information terminal 1A receives wireless signals, measures the amount of movement and rotation of its own terminal, and measures the distance to an external object, similar to step S10 described in Fig. 9. Then, in step S33A, it transmits and receives information on the amount of movement and rotation with mobile information terminal 1D. Then, in step S34A, it combines the information on the amount of movement and the amount of rotation of mobile information terminal 1D and estimates the terminal position of mobile information terminal 1D by processing similar to step S3 described in Fig. 5.

[0082] The next steps S35A and S36A are processes for improving the estimation accuracy by using the position estimation data of the own terminal as seen from the other terminal. That is, when the new position of the other terminal is estimated, an inter-terminal vector V1 is obtained with the own terminal position as the starting point and the other terminal position as the end point. Similarly, an inter-terminal vector V2 is obtained by the measurement of the other terminal. If these two vectors are converted into values ​​of the same coordinate system, they should be inverse vectors if there is no error in the position estimation. Note that the relationship between the coordinate systems is known from data exchange in past position measurements including data on the amount of rotation of each terminal, so the inter-terminal vector measured by the other terminal can be converted into the value of the own terminal. Therefore, in both terminals, the error Δ can be estimated from the sum of these two vectors as shown in the following formula. Δ=(V1+V2) / 2 Therefore, in step S35A, the mobile information terminals transmit and receive estimated position information to obtain the other terminal's position information. Then, in step S36A, the other terminal's position is re-estimated taking into account this error Δ. The re-estimated inter-terminal vector V1' is as follows: V1'=V1-Δ=(V1-V2) / 2 This error correction is for correcting errors caused by the measurement of the amount of movement and rotation of the terminal itself as well as errors caused by the measurement of the radio wave source. If there is no need to improve the estimation accuracy, this re-estimation does not need to be performed. In that case, steps S35A and S36A are skipped.

[0083] Next, in step S37A, the estimated location of the partner terminal and the guide are displayed in the same manner as in step S4 described in Fig. 5. After that, in step S38A, it is determined whether an instruction to end the search has been received, and the processing from steps S32A to S37A is continued until an instruction to end the search is received, and the display of the estimated location of the partner terminal and the display of the guide are continually updated.

[0084] In this way, according to this embodiment, when estimating the location of the other terminal, the location of the own terminal as seen from the other terminal is the location of the other terminal as seen from the own terminal if the vector direction is changed, so by combining the estimation results of both, the estimation accuracy can be improved. In addition, by acquiring the movement history, the past estimation results can also be used as data for current location estimation, and the estimation accuracy can be improved. EXAMPLES

[0085] In this embodiment, an example of using the method in combination with other search methods will be described.

[0086] Fig. 19A is a display example of the estimated position of the radio wave source in this embodiment. In Fig. 19A, when the accuracy of direct estimation of the radio wave source by the radio wave reception of the mobile information terminal is low due to, for example, a small number of data or distance, the mobile information terminal 1 first performs an indirect search, that is, an external reference position estimation of the radio wave source by a positioning technique such as indoor positioning, which is another search method. In Fig. 19A, during the indirect search, a display 97 such as "preparing for direct search" is displayed, and a mark indicating the estimated position of the radio wave source also displays the estimated position of the radio wave source 98, which indicates a wide range that is an approximate position.

[0087] Fig. 19B is a display example of the estimated position of the radio wave source when the accuracy of the estimation of the position of the radio wave source is improved in Fig. 19A. In Fig. 19B, when the estimation accuracy is improved by increasing the number of data or approaching the radio wave source position based on indirect estimation information, the display is switched to the estimated position by direct search. That is, as shown in Fig. 19B, a display 99 indicating the start of direct search is displayed, and the estimated position 81 of the radio wave source with improved estimation accuracy and a guide 82 are displayed, similar to Fig. 2A.

[0088] FIG. 20 is a process flow diagram of radio wave source search in this embodiment. In FIG. 20, the same processes as those in FIG. 5 are assigned the same reference numerals, and the description thereof will be omitted. In FIG. 20, when the mobile information terminal 1 receives a search start instruction in step S1, in step S15, the mobile information terminal 1 estimates the position of the radio wave source by indirect search, which is another search method, and displays the estimated position of the radio wave source showing a wide range that is an approximate position, as shown in FIG. 19A. Then, in step S2 similar to FIG. 5, the mobile information terminal 1 receives a wireless signal by direct search and measures the amount of movement and rotation of the terminal itself. Then, in step S3, the mobile information terminal 1 estimates the position of the radio wave source, and in step S16, it is determined whether the estimated position error is smaller than a predetermined value. If it is not smaller, it is determined that the estimation accuracy is low, and the process returns to step S15, and the mobile information terminal 1 repeats the estimation of the radio wave source position by indirect search until the estimation accuracy reaches the predetermined accuracy. Then, when the estimation accuracy reaches a predetermined accuracy or higher, steps S2 and S3 receive wireless signals, measure the amount of movement and rotation of the terminal itself, and update the estimate of the radio wave source position, and in step S4 switch to displaying the estimated position by direct search, displaying the estimated radio wave source position 81 and guide 82 shown in Fig. 19B. Then, until an instruction to end the search is received, the processing from steps S2 to S4 is continued, and the display of the estimated radio wave source position and the display of the guide are continually updated.

[0089] In this way, in this embodiment, since the estimated location can be estimated even at a stage where the accuracy of the location estimation of the radio wave source by direct search is low, the search can be performed efficiently. This method is also effective for meeting people. That is, first, you head to the meeting place based on map information, and when you get close enough to receive radio waves, you can use the final accurate location estimation by radio wave source search to meet the target person more efficiently.

[0090] Although the embodiments have been described above, the above embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. [Explanation of symbols]

[0091] 1, 1A to 1D: mobile information terminal, 2: server, 9: communication network, 10: housing, 11: display surface, 101: processor, 102: memory, 103: display device, 107: operation input unit, 80: radio wave source, 81: estimated position of radio wave source, 82: guide, 87: first estimated position of radio wave source, 88: second estimated position of radio wave source, 91: guide, 94: estimated position of counterpart terminal, 96: movement history

Claims

1. A portable information terminal including a wireless receiving device for receiving a wireless signal from a radio wave source, a sensor for measuring an amount of movement and an amount of rotation of the terminal itself, a display device, and a control device, The control device includes: estimating a position of the radio wave source from the reception strength of the radio wave source received by the wireless receiving device and the amount of movement and rotation of the terminal itself measured by the sensor; A portable information terminal characterized in that a graphical guide display showing three-dimensional direction and distance showing the relative positional relationship of the estimated position of the radio wave source as seen from the position of the portable information terminal is displayed on the display device superimposed on an external world image.

2. 2. The portable information terminal according to claim 1, A portable information terminal characterized in that a tip of the graphical guide display overlaps with a display of the estimated position of the radio wave source.

3. 2. The portable information terminal according to claim 1, A portable information terminal characterized in that the leading edge of said graphical guide display does not overlap with the display of the estimated position of said radio wave source.

4. 2. The portable information terminal according to claim 1, A portable information terminal, wherein the graphical guide display also reflects an obstruction relationship with actual objects in the outside world.

5. 2. The portable information terminal according to claim 1, A portable information terminal characterized in that the calculation of the estimated position of the radio wave source is performed by a server.

6. 2. The portable information terminal according to claim 1, The portable information terminal is characterized in that the control device displays an error range of the estimated position of the radio wave source including differences due to three-dimensional directions.

7. 7. The portable information terminal according to claim 6, The portable information terminal is characterized in that the control device displays guidance for reducing an error in the estimated position of the radio wave source.

8. 2. The portable information terminal according to claim 1, The portable information terminal is characterized in that the control device displays estimated positions of a plurality of radio wave sources and changes the manner in which the graphical guide display is displayed depending on the distance difference between the estimated positions of the plurality of radio wave sources.

9. 2. The portable information terminal according to claim 1, The portable information terminal is characterized in that the control device also displays the direction of movement to the estimated position of the radio wave source.

10. 2. The portable information terminal according to claim 1, The location of the radio wave source is estimated by another portable information terminal; The control device measures a positional relationship with the other portable information terminal, and based on the information on the positional relationship, converts the estimated position of the radio wave source transmitted from the other portable information terminal into an estimated position of the radio wave source in the coordinate system of the portable information terminal itself, and displays the graphical guide.

11. 2. The portable information terminal according to claim 1, The graphical guide display is performed on another portable information terminal, The portable information terminal according to claim 1, wherein the control device transmits the estimated position of the radio wave source to the other portable information terminal which is causing the other portable information terminal to display the graphical guide.

12. 2. The portable information terminal according to claim 1, The portable information terminal is characterized in that the control device displays the graphical guide display by superimposing it on an image from the user's viewpoint that includes the radio wave source.

13. 2. The portable information terminal according to claim 1, The control device is characterized in that it receives information from another portable information terminal that estimates the position of the portable information terminal itself, and displays the graphical guide for the estimated position of the other portable information terminal by referring to the received information.

14. 2. The portable information terminal according to claim 1, The portable information terminal is characterized in that the control device also displays a movement history of the position of the radio wave source.

15. 2. The portable information terminal according to claim 1, The control device displays the results of position estimation by another method prior to estimating the position of the radio wave source, and displays the graphical guide when the accuracy of the estimation of the position of the radio wave source reaches a predetermined level or higher.

16. A display method for a mobile information terminal that estimates and displays a position of a radio wave source, comprising: Estimating the position of the radio wave source from the received signal strength of the radio wave source and the amount of movement and rotation of the terminal; A display method for a portable information terminal, characterized in that a graphical guide display showing three-dimensional direction and distance showing the relative positional relationship of the estimated position of the radio wave source as seen from the position of the portable information terminal is displayed superimposed on an external image.

17. A display method for a portable information terminal according to claim 16, comprising: The location of the radio wave source is estimated by another portable information terminal; Measure a positional relationship with the other portable information terminal; A display method for a mobile information terminal, characterized in that, based on the information on the positional relationship, the estimated position of the radio wave source transmitted from the other mobile information terminal is converted into an estimated position of the radio wave source in the coordinate system of the mobile information terminal itself, and the graphical guide display is performed.

Images