Location estimation system, information terminal used therein, and location estimation method

Abstract

To provide a location estimation system with improved location accuracy using an information terminal.SOLUTION: A location estimation system includes a first information terminal having a function of transmitting a wireless signal accompanied by identification information and a second information terminal having a function of measuring the reception strength of the wireless signal. The second information terminal measures the reception strength of the wireless signal transmitted by the first information terminal. At least one of the first information terminal and the second information terminal measures a traveled distance. At least one of the first information terminal and the second information terminal estimates a relative positional relation between the first information terminal and the second information terminal from a change in the reception strength of the wireless signal with respect to the traveled distance.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a position estimation system using an information terminal. [Background technology]

[0002] Background art in the technical field related to obtaining location information is disclosed in Patent Document 1. In Patent Document 1, the user's location is estimated from the location of a nearby wireless LAN (Local Area Network) access point or measured using a GPS (Global Positioning System). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-351859 Summary of the Invention [Problem to be solved by the invention]

[0004] The location estimation method of Patent Document 1 may not be able to achieve sufficient accuracy in location estimation in some cases. For example, this may occur when GPS radio waves are disrupted by reflections from buildings, when reception is not possible inside a building, when the user is far away from a wireless LAN access point, or when radio waves from the wireless LAN access point cannot be received. High location estimation accuracy is desirable in cases such as when calling a taxi in a place with heavy pedestrian and vehicular traffic, where the appearance of the person calling is unknown and there are multiple possible people or vehicles.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide a position estimation system using an information terminal with improved position accuracy. [Means for solving the problem]

[0006] One example of the present invention is a position estimation system comprising a first information terminal having the function of transmitting a radio signal accompanied by identification information and a second information terminal having the function of measuring the reception strength of the radio signal, wherein the second information terminal measures the reception strength of the radio signal transmitted by the first information terminal, at least one of the first information terminal and the second information terminal measures the travel distance, and at least one of the first information terminal and the second information terminal estimates the relative positional relationship between the first information terminal and the second information terminal from the change in the reception strength of the radio signal with respect to the travel distance. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a position estimation system using an information terminal with improved position accuracy. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic configuration diagram of a position estimation system according to a first embodiment. [Figure 2] 10A and 10B are diagrams illustrating a meeting support operation in a situation where a passenger is waiting for a vehicle such as a taxi on the side of the road in the first embodiment. [Figure 3A] 10 is a diagram showing the positional relationship between information terminal 1A and information terminal 1B in the first embodiment when information terminal 1B does not move. FIG. [Figure 3B] 10 is a diagram showing the positional relationship between information terminal 1A and information terminal 1B in the first embodiment when information terminal 1B also moves. FIG. [Figure 3C] 3A and 3B. This is a projection diagram in which each element is projected in the direction of the movement line 110A onto a vertical plane perpendicular to the movement line 110A in FIGS. 3A and 3B. [Figure 4A] 10 is a profile of the reception intensity P0 when there is no reflection relative to the relative movement distance x in the first embodiment. [Figure 4B] 10 is a profile of the reception intensity P including the reflected wave with respect to the relative movement distance x in the first embodiment. [Figure 5A] 10 is a diagram showing how the estimated value x0E of the closest travel distance changes in the first embodiment. FIG. [Figure 5B]10 is a diagram showing measured values of reception intensity P including reflected waves with respect to relative movement distance x in Example 1. FIG. [Figure 6A] 10A and 10B are diagrams illustrating an example of control of acceleration change with respect to position information in the first embodiment. [Figure 6B] 10A and 10B are diagrams illustrating an example of controlling a speed change in response to position information in the first embodiment. [Figure 7] 1 illustrates an example of the external configuration of an HMD as an example of an information terminal according to a first embodiment. [Figure 8] 8 is a functional block configuration example of the information terminal (HMD) of FIG. 7. [Figure 9] FIG. 10 is a control flow diagram for meeting support in the first embodiment. [Figure 10A] FIG. 10 is a diagram illustrating internal data in which reception intensity is recorded with respect to measurement time in the first embodiment. [Figure 10B] FIG. 10 is a diagram illustrating internal data in which travel distances are recorded relative to measurement times in the first embodiment. [Figure 10C] 10 is a diagram illustrating internal data in which reception intensity is recorded with respect to relative movement distance in the first embodiment. FIG. [Figure 10D] 10 is a diagram illustrating internal data that records an estimated value of the closest movement distance for the relative movement distance in the first embodiment. FIG. [Figure 11A] FIG. 10 is an explanatory diagram showing an estimated distance to a partner information terminal displayed on a display screen of an information terminal in the second embodiment. [Figure 11B] FIG. 10 is an explanatory diagram showing an AR object mark displayed at an estimated position of a partner information terminal on a display screen of an information terminal in the second embodiment. [Figure 11C] FIG. 10 is an explanatory diagram showing an AR object mark representing the partner information terminal itself displayed on the display screen of the information terminal in the second embodiment. [Figure 12A] 10 is a diagram showing the positional relationship of each element on a vertical plane including an information terminal 1B and perpendicular to a movement line 110A in the second embodiment. FIG. [Figure 12B] 12B is a diagram showing the positional relationship when each element in FIG. 12A is projected vertically onto the ground surface 111. FIG. [Figure 13A]FIG. 10 is an explanatory diagram of a hardware configuration on the vehicle side in a third embodiment. [Figure 13B] FIG. 11 is an explanatory diagram of another configuration of the vehicle-side hardware in the third embodiment. [Figure 14A] FIG. 11 is an explanatory diagram showing the estimated distance of a passenger displayed on the display screen of an information terminal in the third embodiment. [Figure 14B] FIG. 11 is an explanatory diagram showing an AR object mark displayed at an estimated position of a passenger on the display screen of an information terminal in the third embodiment. [Figure 14C] FIG. 11 is an explanatory diagram showing an AR object mark indicating a passenger displayed on the display screen of the information terminal in the third embodiment. [Figure 15A] 11 is a diagram showing the positional relationship of each element on a vertical plane including an information terminal 1A in the third embodiment and perpendicular to a movement line 110B. FIG. [Figure 15B] 15B is a diagram showing the positional relationship when each element in FIG. 15A is projected vertically onto the ground surface 111. FIG. [Figure 16] FIG. 10 is an explanatory diagram for explaining a method for evaluating the influence of buildings that contribute to the reflection of radio waves in the fourth embodiment. [Figure 17A] FIG. 11 is an explanatory diagram for switching fitting functions depending on the amount of wall surfaces of a building in the fourth embodiment. [Figure 17B] FIG. 11 is an explanatory diagram for switching fitting functions depending on the distance to the meeting place in the fourth embodiment. [Figure 17C] FIG. 13 is an explanatory diagram for switching a predetermined distance TP for switching a fitting function depending on the amount of wall surfaces of a building in the fourth embodiment. [Figure 18A] FIG. 20 is a diagram illustrating how a vehicle approaching in front at the time when the reception strength reaches a maximum is photographed as a waiting vehicle in the sixth embodiment. [Figure 18B] 13 is a diagram illustrating how an image captured from a frontal position is displayed on the screen of an information terminal and a mark is superimposed on the image of the target vehicle in Example 6. FIG. [Figure 19A] 13 shows a display screen of information terminal 1C in a state where the other party's information terminal 1A is still far from the meeting place and information terminal 1C is not receiving a wireless signal in the seventh embodiment. [Figure 19B] 13 shows a display screen of information terminal 1C in a state where portable information terminal 1A in the seventh embodiment is approaching the meeting place and information terminal 1C is receiving a wireless signal. [Figure 20] FIG. 20 is an explanatory diagram showing how the display is changed according to the reception strength in the seventh embodiment. [Figure 21] 13 is a display screen displaying an explanatory message indicating that the position is unknown in the seventh embodiment. [Figure 22] 13 is a display screen displaying a service switching input box in the seventh embodiment. [Figure 23] FIG. 20 is an explanatory diagram of a voice notification corresponding to the reception strength of a wireless signal in the seventh embodiment. [Figure 24] FIG. 13 is an explanatory diagram of vibration notification corresponding to the reception strength of a wireless signal in the seventh embodiment. [Figure 25] 13 is a display screen of an information terminal 1A in the seventh embodiment. [Figure 26A] 13 is a display example of a partner information terminal image of an information terminal 1C in the eighth embodiment. [Figure 26B] 13 is a display example in which an image of a partner information terminal is superimposed on a map display screen of an information terminal 1C in the eighth embodiment. [Figure 27] FIG. 20 is a diagram illustrating a modified example of the person identification method in the eighth embodiment. [Figure 28] 13 is a display example of a recognition result of an information terminal 1C by a counterpart information terminal 1A in the eighth embodiment. [Figure 29] 13 is a display format for listing person images for correcting a person to be met in the eighth embodiment. [Figure 30] 13 is a display screen of an information terminal 1A in the eighth embodiment. [Figure 31] FIG. 20 is a control flow diagram for meeting support in the eighth embodiment. [Figure 32] 13 is a display screen of an information terminal 1B in the ninth embodiment. [Figure 33] FIG. 13 is an explanatory diagram of transmission information of an optical signal of a light-emitting device according to a ninth embodiment. [Figure 34]FIG. 13 is a control flow diagram for meeting support in the ninth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Example]

[0010] In this embodiment, an example of a location estimation system will be described in which a passenger calls a taxi and provides meeting support to confirm the other party. In the following description, an information terminal may be simply referred to as a terminal. Also, meeting support may be simply referred to as support.

[0011] Fig. 1 is a schematic diagram of a location estimation system according to this embodiment. In Fig. 1, a wireless signal source carrying identification information is held by an information terminal 1A in a vehicle. The vehicle itself may be the information terminal 1A, or the information terminal 1A may be mounted on the vehicle.

[0012] A passenger has at least one of the following information terminals: information terminal 1B, which is an HMD (Head Mount Display), information terminal 1C, which is a smartphone, and information terminal 1D, which is a smartwatch. The following operations of the passenger side terminals may be performed by these terminals independently or in cooperation with each other. Here, an example is shown in which information terminal 1B, which is an HMD, operates independently as the passenger side terminal. Furthermore, information terminals 1A to 1D are collectively referred to as information terminal 1.

[0013] The server 2 performs processing on behalf of each information terminal 1, mediates the sending and receiving of information between each information terminal 1, and provides necessary information via the communication network 9. The server 2 may be, for example, a local server, a cloud server, an edge server, an internet service, or the like, and its form is not important.

[0014] FIG. 2 is a diagram illustrating the meeting support operation in this embodiment when a passenger is waiting for a vehicle such as a taxi on the side of the road. In FIG. 2, information terminal 1A is the vehicle itself or an information terminal mounted on the vehicle. Information terminal 1A moves along movement path 110A on the road and approaches information terminal 1B carried by the passenger. Here, movement path 110A refers to the path that information terminal 1A has taken in the past and the path that it is scheduled to take in the future. For the sake of explanation, the passenger is initially assumed to be stationary, but a generalized case of the passenger moving along movement path 110B on the side of the road will also be described later.

[0015] The information terminal 1A continues to transmit a wireless signal accompanied by identification information while moving. Here, the wireless signal accompanied by identification information may be any signal that can be identified by directly transmitting and receiving a wireless signal without going through a repeater or the like. For example, the wireless signal may be a beacon signal, or a signal for Bluetooth (registered trademark), BLE (Bluetooth low energy), Wi-Fi (registered trademark), or local 5G point-to-point communication. Furthermore, this wireless signal may be used to communicate between the information terminals 1.

[0016] A moving information terminal 1 measures its own position. Measurement may be performed using a positioning system that uses radio waves such as GPS, a positioning system based on a position reference point, or by integrating the output of an acceleration sensor. The position may be recorded as the distance traveled from an appropriately determined starting point for each time. It is also possible to calculate speed from data on the distance traveled for each time. Hereinafter, the position of information terminal 1A will be expressed as the distance traveled from the starting point. When this position information is used by the other information terminal 1B, it is provided from information terminal 1A to information terminal 1B. Meanwhile, information terminal 1B receives the wireless signal and measures the reception strength.

[0017] When both information terminals 1 move, the change in relative movement distance is used from the movement distance of both. Either information terminal may be the transmitter of the wireless signal. Furthermore, both may transmit and receive wireless signals.

[0018] FIG. 3A is a diagram showing the positional relationship between information terminal 1A and information terminal 1B in this embodiment when information terminal 1B is not moving. FIG. 3A is a projection diagram in which each element is projected vertically onto a horizontal plane, and hereafter, unless otherwise specified, when referring to a projection diagram onto a horizontal plane, it means a diagram in which each element is projected vertically. Furthermore, the symbols for each element may be the same as those used on the projection diagram before projection. Furthermore, movement lines 110A and 110B are approximated by straight lines. Furthermore, only the positions of information terminals 1A and 1B are shown.

[0019] In FIG. 3A, the travel distance 114A from the starting point 112A of the information terminal 1A is expressed as x A The starting point 112A may be determined arbitrarily. For example, the position of information terminal 1A at the time when information terminal 1A and information terminal 1B start communication when meeting is defined as starting point 112A. Even if a point that has never actually been there is defined as starting point 112A, the distance traveled from starting point 112A is expressed here as travel distance 114A.

[0020] Also, the distance that information terminal 1A must travel from the starting point (closest travel distance) until it reaches the position (closest position) on movement line 110A closest to information terminal 1B is denoted as x0. Before arriving at the closest position, the closest position is estimated from data on the reception strength of the wireless signal relative to the movement distance (details will be described later). Also, the distance 113 between information terminal 1B and movement line 110A is denoted as d.

[0021] FIG. 3B is a diagram showing the positional relationship between information terminal 1A and information terminal 1B in this embodiment when information terminal 1B also moves, and is a projection onto a horizontal plane, similar to FIG. 3A.

[0022] In Fig. 3B, information terminal 1B moves along movement line 110B in a direction approaching information terminal 1A. Let x be the movement distance 114B of information terminal 1B along movement line 110B from an arbitrarily determined starting point 112B. BThe movement line 110A and the movement line 110B are approximately parallel. The distance between the starting point 112A and the starting point 112B is the closest movement distance x0. Note that the movement distance here is a quantity used to represent the distance from the starting point, and depending on how the starting point is defined, it may not necessarily match the distance that the information terminal 1 actually moves while the meeting support operation is being performed.

[0023] FIG. 3C is a projection diagram in which each element is projected in the direction of the movement line 110A onto a vertical plane perpendicular to the movement line 110A in FIGS. 3A and 3B. Hereinafter, unless otherwise specified, when the expression "projection diagram onto a vertical plane perpendicular to the movement line" is used, it means a diagram in which each element is projected in the direction of the movement line. This diagram is common to both the case where the information terminal 1B moves and the case where it does not move. The ground plane 111 is approximated as a horizontal plane. The distance 115A between the information terminal 1A and the ground plane 111 is defined as h A and the distance 115B between the information terminal 1B and the ground surface 111 is expressed as h B It is written as follows.

[0024] In the following discussion, it is generalized and assumed that the information terminal 1B also moves. The relative movement distance x is defined by the following equation (1), and the profile of the received signal strength with respect to the relative movement distance is considered. x=x A +x B …(1) If the information terminal 1B does not move, press x. B = 0.

[0025] 3C, 120 indicates the path of the direct wave of the wireless signal, and 121 indicates the path of the reflected wave reflected by the ground surface 111. The received wireless signal includes signals reflected by the ground surface and nearby buildings, but first, consider the profile of the received signal strength that does not include reflected waves.

[0026] The received strength P0 of a wireless signal (direct wave) when there is no reflection is inversely proportional to the square of the distance between the transmission source (information terminal 1A) and the receiver (information terminal 1B), and is therefore expressed by the following equation (2). P0=k0 / {(x-x0) 2 +g0 2} …(2) Here, k0 is a constant that depends on the signal source strength and the receiver sensitivity, and g0 is defined by the following equation (3). g0={d 2 +(h A -h B ) 2} 1 / 2 …(3) g0 is the closest distance in the direct wave path 120;

[0027] For example, h A and h B When the values are equal, d is 5m, and x0 is 50m, the profile of P0 is as shown in Figure 4A. g0 is 5m. Here, normalization is performed so that P0 = 1 at the closest position (x = x0). As can be seen in Figure 4A, P0 has a profile with a sharp peak at the closest position.

[0028] In reality, the reception intensity profile differs from P0 because there are components reflected by the ground surface 111 and nearby buildings. The position at which the reception intensity of the reflected component is greatest depends on the position of the reflector, but on average it can be expected to be near the closest position. Furthermore, since the path of the reflected wave is longer than the direct wave path 120, when the reflected wave is also taken into account, the profile effectively becomes such that the closest distance is longer. Therefore, the reception intensity profile including the reflected wave will also be explained by approximating it using the following equation (4). P=k / {(x-x0) 2 +g 2} …(4) Here, k is a constant that depends on the signal source strength and receiver sensitivity, and g is the effective closest distance. If this effective closest distance is the closest distance of the reflection path 121 that is reflected by the ground surface 111, which is likely to be the strongest reflected wave, then the following equation (5) is obtained. g={d 2 +(h A +h B ) 2} 1 / 2 …(5) Here, d=5m, h A =h B= 1.6 m, then g = 5.9 m, and the closest distance in this case is not significantly different from the closest distance for direct waves. This value of g is calculated as an estimated value (see below), but it can also be used as an approximation of d and g0.

[0029] Figure 4B shows the profile of equation (4). There is no significant difference in the profile compared to Figure 4A. For this reason, we believe that the approximation of the actual received intensity profile, in which the closest distance is replaced with the effective distance in the function form of the direct wave, is appropriate.

[0030] Next, we will explain the position estimation method based on the reception strength profile in Figure 4B. Specifically, the parameters in equation (4) are estimated by parameter fitting using the least squares method or the like from the measured reception strength while approaching the closest position. As an example, the reciprocal of the reception strength is a quadratic function of the relative movement distance x, so fitting is performed using this function. 1 / P=(1 / k){(x-x0) 2 +g 2}=Ax 2 +Bx+C …(6) Equation (6) is linear with respect to parameters A, B, and C, and is calculated from the received signal strength data by the least squares method. Here, when calculating by the least squares method, the measurement error of the received signal strength P is assumed to be constant, and the error of 1 / P is assumed to be P -2 Weighted processing may be performed assuming that the parameter is proportional to the value of the parameter. This allows the estimated values of each parameter in equation (4) to be calculated as follows: k=1 / A …(7) x0=-B / (2A) …(8) g=(4AC-B 2 ) 1 / 2 / (2A) …(9) The parameter fitting method is not limited to the above method.

[0031] The values of the parameters k, x0, and g are updated as measurement data of the received signal strength is added while moving, and errors due to noise are reduced.

[0032] The estimated value of the nearest moving distance x0 is recalculated as x0E Let x 0E The change in the received signal strength P is shown in Figure 5A. Figure 5B shows the measured received signal strength 200 and the received signal strength profile 201 estimated by fitting. For example, the point at x=30 m on the graph in Figure 5A is the x value estimated from the measured received signal strength P up to x=30 m shown in Figure 5B. 0E It is not necessary to use all past measurements for estimation. When approaching the nearest position and the accumulated number of measurements with small errors increases, the measurement values with large errors at the beginning of measurement are used as x 0E It may be possible to not use it in estimating

[0033] Estimated value x from received signal strength 0E While the noise influence on x is large, the vehicle will control its speed to target the location specified at the time of reservation. 0E After the noise impact on the vehicle has decreased, the vehicle controls its speed to stop at the closest position to the passenger. The vehicle can be controlled by the driver or an automatic driving device.

[0034] The control that is desirable for the vehicle control device is as follows: First, information (x) about the position where the reception strength of the wireless signal is the maximum is received from the information terminal 1A. 0E ) and the location information of both information terminals 1 (x A , x B ) is provided. Based on this information, the system controls the vehicle to stop by monotonically decreasing the magnitude of the speed and acceleration for a continuous period up to and including the point of stopping, with the goal of stopping at the closest position. This allows the vehicle to stop accurately at the closest position, while also reducing the magnitude of acceleration when stopping, thereby reducing the shock felt by passengers when stopping. Figures 6A and 6B show examples of control. Figure 6A shows the acceleration change, and Figure 6B shows the speed change.

[0035] An example of speed control is shown below, assuming that a passenger approaches at a constant speed. In order to reduce the shock when the car stops, the speed is controlled with the goal of making the acceleration of the car zero at the same time. Therefore, as an example, it is assumed that the moving speed of the passenger is constant from a certain point onwards, and the acceleration of the car is kept constant, and the car control device sets the acceleration at that point so that the speed and acceleration of the car become zero at the closest moving distance position. If the speed of the information terminal 1A is v A , acceleration is a A Let the speed of information terminal 1B be v B Then, a A Specifically, the setting is as follows: a A =-2v A (v A / 3+v B ) / (x 0E -x A -x B ) …(10) This acceleration setting allows the vehicle controller to control the vehicle speed.

[0036] At this time, the distance the car travels to the nearest position is x R (hereafter referred to as the target distance) is x R =(x 0E -x A -x B ) / (1+3v B / v A ) …(11) This is expressed as equation (11).

[0037] When passengers are stopped, B = 0. Also, the speed of the passenger is not always constant, and x 0E Since the value of may also change, the value of the right side of equations (10) and (11) is obtained at each time.

[0038] When the driver controls the vehicle, the information terminal 1A detects the target distance x R The driver may then use this value as a guide to control the speed.

[0039] For safety reasons, the acceleration a is set to prevent sudden acceleration or deceleration. A A predetermined upper limit may be set for the absolute value of

[0040] Here, there may be estimation errors or prediction errors, such as when the user with information terminal 1B initially stops but starts approaching as information terminal 1A approaches. In this case, the reception strength profile is checked, and if the peak value is exceeded, the car should be stopped immediately by controlling the speed within a safe range.

[0041] As a result, it is possible to estimate the location of the meeting place more precisely based on the reception strength of the wireless signal.

[0042] Fig. 7 shows an example of the external configuration of an HMD as an example of an information terminal in this embodiment. In Fig. 7, the HMD is provided with a display device including a display surface 11 in a glasses-like housing 10. This display device is, for example, a transmissive display device, and a real image of the outside world is transmitted to the display surface 11, and an image is superimposed on the real image. A controller, a camera 12, a distance measurement sensor 13, other sensor units 14, etc. are mounted on the housing 10.

[0043] The camera 12 has, for example, two cameras arranged on the left and right sides of the housing 10, and captures an image by capturing a range including the front of the HMD. The distance measurement sensor 13 is a sensor that measures the distance between the HMD and an object in the external world. The distance measurement sensor 13 may use a TOF (Time Of Flight) sensor, a stereo camera, or another method. The sensor unit 14 includes a group of sensors for detecting the position and orientation of the HMD. On the left and right sides of the housing 10, there are provided an audio input device 18 including a microphone, an audio output device 19 including a speaker and an earphone jack, and the like.

[0044] The information terminal 1 may be provided with an operating device 20 such as a remote controller. In this case, the HMD performs, for example, short-range wireless communication with the operating device 20. By operating the operating device 20 with the user's hand, the user can input instructions related to the functions of the HMD, move a cursor on the display surface 11, and the like. The HMD may communicate with an external smartphone, PC, or the like to cooperate with the HMD. For example, the HMD may receive AR (Augmented Reality) image data from a smartphone application.

[0045] The information terminal 1 may display a virtual image such as an AR image on the display surface 11. For example, the information terminal 1 generates a virtual image for guiding the user and displays it on the display surface 11.

[0046] Fig. 8 is a functional block diagram of the information terminal 1 (HMD) of Fig. 7. Basically, other information terminals 1 have the same configuration. The information terminal 1 includes a processor 101, a memory 102, a camera 12, a distance measurement sensor 13, a sensor unit 14, a display device 103, a communication device 104, an audio input device 18 including a microphone, an audio output device 19 including a speaker or the like, a vibration generator 1100 such as a vibrator, a light-emitting device 3000, a wireless transmitting device 105, a wireless receiving device 106, an operation input unit 107, and a battery 108. These elements are connected to each other via a bus or the like.

[0047] The processor 101 is composed of a CPU, ROM, RAM, etc., and constitutes the controller of the HMD. The processor 101 executes processes in accordance with the control program 31 and application program 32 in the memory 102, thereby realizing functions of the OS, middleware, application, etc., and other functions. The memory 102 is composed of a non-volatile storage device, etc., and stores various data and information handled by the processor 101, etc. The memory 102 also stores, as temporary information, images acquired by the camera 12, etc., and detection information, etc.

[0048] The camera 12 acquires an image by converting light incident through a lens into an electrical signal using an imaging element. When the distance measurement sensor 13 uses, for example, a TOF (Time Of Flight) sensor, it calculates the distance to an object from the time it takes for light emitted into the outside world to hit the object and return. The sensor unit 14 includes, for example, an acceleration sensor 141, a gyro sensor (angular velocity sensor) 142, a geomagnetic sensor 143, and a GPS receiver 144. The sensor unit 14 detects the position, orientation, movement, and other conditions of the HMD using information detected by these sensors. The HMD is not limited to these, and may also include an illuminance sensor, a proximity sensor, a barometric pressure sensor, and the like.

[0049] The display device 103 includes a display drive circuit and a display surface 11, and displays a virtual image or the like on the display surface 11 based on the image data of the display information 34. Note that the display device 103 is not limited to a transmissive display device, and may be a non-transmissive display device or the like.

[0050] The communication device 104 includes a communication processing circuit, an antenna, and the like corresponding to various predetermined communication interfaces. Examples of the communication interface include a mobile network, Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared, and the like. The communication device 104 performs wireless communication processing with other information terminals 1 and access points. The communication device 104 also performs short-range communication processing with a controller.

[0051] The wireless transmitting device 105 transmits a wireless signal accompanied by an identification signal used for position estimation. Here, the wireless signal accompanied by identification information is, for example, a Bluetooth signal. The wireless receiving device 106 receives the wireless signal accompanied by the identification signal and measures the reception strength. Information communication may be performed using the wireless transmitting device 105 and the wireless receiving device 106.

[0052] In addition, in the case of an information terminal 1 that only transmits or only receives wireless signals, it is sufficient if it is equipped with the necessary devices.

[0053] The audio input device 18 converts audio input from a microphone into audio data. The audio output device 19 outputs audio from a speaker or the like based on the audio data. The audio input device may have a voice recognition function. The audio output device may have a voice synthesis function.

[0054] The vibration generator 1100 has a function of generating vibrations to notify the user. The light-emitting device 3000 has a function of transmitting a code using intensity-modulated light. Note that near-infrared light may be used so as to be unobtrusive.

[0055] The operation input unit 107 is a part that accepts operation inputs to the HMD, such as power on / off, volume adjustment, etc., and is composed of hardware buttons, touch sensors, etc. The battery 108 supplies power to each unit.

[0056] The controller by the processor 101 has, as an example of a configuration of functional blocks realized by processing, a communication control unit 101A, a display control unit 101B, a data processing unit 101C, and a data acquisition unit 101D.

[0057] The memory 102 stores a control program 31, an application program 32, setting information 33, display information 34, position estimation information 35, etc. The control program 31 is a program for realizing estimation of the relative positional relationship between information terminals 1. The application program 32 is a program for realizing a guidance function for the user. The setting information 33 includes system setting information and user setting information related to each function. The display information 34 includes image data and position coordinate information for displaying a virtual image on the display surface 11. The position estimation information 35 is information related to the movement distance of the information terminal 1 and the reception strength of a wireless signal for performing position estimation.

[0058] The communication control unit 101A controls communication processing using the communication device 104, such as when communicating with another information terminal 1. The display control unit 101B uses the display information 34 to control the display of a virtual image or the like on the display surface 11 of the display device 103.

[0059] The data processing unit 101C reads and writes the position estimation information 35, and estimates the relative positional relationship between the own terminal and the other terminal.

[0060] The data acquisition unit 101D acquires the strength of the wireless signal from the wireless receiving device 106, and acquires each detection data from various sensors such as the camera 12, the distance measurement sensor 13, and the sensor unit 14. The data acquisition unit 101D estimates its own position and measures the movement distance from the detection data of the various sensors.

[0061] FIG. 9 is a control flow diagram of rendezvous support in this embodiment. In FIG. 9, before this flow, the approximate location of the rendezvous is decided, and the receiving information terminal 1 acquires the identification information of the wireless signal. Either information terminal may be the transmitting side, and wireless signals may be transmitted from both. When both information terminals 1 enter the vicinity of the rendezvous place, first, in steps S1A and S1B, communication between the information terminals 1 is established, and a trigger is set to start rendezvous support. In addition, the moving information terminal 1 sets starting points 112A and 112B for measuring the travel distance. This communication may be via a communication network, or may be direct communication between the information terminals 1. After this, a position estimation loop is entered.

[0062] Within the position estimation loop, first, in steps S2A and S2B, the transmitting information terminal 1 transmits a wireless signal, and the receiving information terminal 1 receives the wireless signal and measures the reception strength. As a result, the receiving information terminal 1 generates internal data that records the reception strength at the measurement time, as shown in FIG. 10A, for example. When both information terminals transmit wireless signals, two radio wave strength profiles are obtained as a result. Accordingly, two types of position estimation are possible, and both estimation results may be used, or only one result may be used. Furthermore, the moving information terminal 1 measures the travel distance and generates travel distance data, as shown in FIG. 10B, for example.

[0063] In steps S3A and S3B, each information terminal 1 transmits reception intensity data and movement distance data to the other information terminal 1. Data does not need to be transmitted to information terminals 1 that do not perform position estimation.

[0064] In steps S4A and S4B, the information terminal 1 that performs location estimation generates a reception strength profile for the relative movement distance as shown in Fig. 10C from the reception strength data and movement distance data. Then, the information terminal 1 that performs location estimation generates an estimated value x of the closest movement distance as shown in Fig. 10D, for example, by the method described above. 0E In this case, if both information terminals 1 transmit wireless signals and the other information terminal 1 receives the wireless signals, two types of reception intensity profiles are obtained. In this case, both types may be used, or only one may be used. Note that location estimation in the information terminal 1 starts after at least the amount of data necessary to perform location estimation has been accumulated.

[0065] In steps S5A and S5B, the information terminal 1 that has performed the position estimation transmits the estimated value x of the closest movement distance calculated by the own terminal to the other information terminal 1. 0E If the other terminal does not need the information, it does not need to be sent.

[0066] In steps S6A and S6B, each information terminal 1 provides the user or the vehicle control device with information to support the rendezvous. The manner in which this information is provided will be described in detail in the following embodiment.

[0067] In steps S7A and S7B, if both parties can confirm the other party's terminal, i.e., the waiting car or the waiting passenger, and there is no longer any need for meeting support, the meeting support is terminated, but if there is a need, a branch is made to continue the support.

[0068] As described above, according to this embodiment, it is possible to support rendezvous with improved positional accuracy.

[0069] The method of this embodiment is not limited to meetings between people and cars, but can also be applied to meetings between people. In this embodiment, the wireless signal source with identification information is described as being held by an information terminal, but it is also possible for the device to have only a wireless signal source without the functionality of an information terminal. For example, this method can be used to support a child who is lost and is holding only a wireless signal source. [Example]

[0070] In this embodiment, the support provided to the user on the passenger side, that is, on the information terminal 1B, will be described.

[0071] The information terminal 1B acquires the information necessary for the support according to the embodiment 1. The support content according to this embodiment is similar to that in the case where the information terminal 1A is mounted in a car and the driver receives support by looking at the display of the information terminal 1A.

[0072] First, as a simple method, the information terminal 1B displays the estimated distance of the other information terminal 1A on the display screen to provide the user with clues to find the car. This is shown in FIG. 11A. In FIG. 11A, 500 is the display screen of the information terminal 1B, and 501 is an explanatory message for support. The estimated distance x of the car (i.e., the other information terminal) when the current position of the information terminal 1B is used as the starting point is d is given by the following equation (12). x d =x 0E -x A -x B …(12) If you are using a see-through HMD, you can see the road conditions as they are. If you are using other information terminals, you can use a camera to take pictures of the road conditions and display them.

[0073] 11B, the information terminal 1B may display an AR object mark 502 at the estimated position of the car, and may simultaneously display an explanatory message 503.

[0074] Furthermore, if there are no other candidate cars or if the car of the person to be met can be identified from the characteristics of the car (model, color, license plate number, etc.), information terminal 1B may display AR object mark 504 representing the car itself, as shown in Fig. 11C. An explanatory message 505 may also be displayed at the same time. Furthermore, if there are multiple candidate cars, AR object marks 504 may be displayed on all of the candidate cars.

[0075] Furthermore, when communication between the information terminals is started or when it is estimated that the distance between the two terminals is less than a predetermined distance, the information terminal 1B may notify the user by vibrating, etc. By doing so, the user does not need to pay attention to the location of the car until it gets close.

[0076] The above is an example of support for a passenger, but similar support can also be provided for a car driver (details will be described later). In that case, the image on the display of the information terminal 1A will be that of the passenger. Also, characteristic information of the passenger (for example, a photo) may be used to assist in the estimation.

[0077] Furthermore, information about the characteristics of the exterior of the passenger or vehicle acquired from the server may also be used in estimating the position on the display screen.

[0078] In order to display the marks as in the cases of Figs. 11B and 11C, the estimated distance x d It is necessary to estimate the position on the road corresponding to the location of the vehicle. Specifically, the position is estimated as follows. Note that the following estimation method is also applicable to estimating the passenger's position from the vehicle side. In this case, each geometric element is replaced with the geometric element of the other side.

[0079] First, information terminal 1B measures the distance d (113 in FIG. 3A) between itself and the moving line 110A (FIG. 3A) of information terminal 1A. B (115B in FIG. 3C) is assumed to be known by the user's settings etc. B Instead of the value of h, the value of h is calculated from the user's height data recorded on the information terminal 1 or the server. BThe value of may be estimated.

[0080] Distance d is estimated as shown in FIG. 12A. FIG. 12A is a diagram including information terminal 1B, showing the positional relationship of each element on a vertical plane perpendicular to movement line 110A. Movement line 110A is projected vertically onto ground surface 111, and a projection line 130A is defined as the line. Projection line 130A cannot be accurately determined unless the vehicle is identified, but since it is assumed that the vehicle will be traveling in the center of the lane, there will be no significant error, and therefore the center line of the lane is defined as projection line 130A. Information terminal 1B estimates that a perpendicular line 131B is drawn from its own terminal to projection line 130A, and calculates angle 133B (φ B The angle φ is measured based on the camera image and the attitude information of the information terminal 1B. B Once this is known, the distance d is d=h B tanφ B …(13) It is calculated using equation (13).

[0081] If this distance d is known, the relationship between a point along the projection line 130A and the direction of that point can be determined as follows. FIG. 12B is a diagram showing the positional relationship of each element projected vertically onto the ground surface 111. In FIG. 12B, point 1B represents the point where the position of information terminal 1B is projected vertically onto the ground surface 111. p 0B (134B) is the foot of the perpendicular line drawn from 1B to the projection line 130A. And, on the ground 111 of FIG. 12B, 1B and p 0B and the estimated position p of the other party's information terminal 1A on the projection line 130A. B The angle (139B) between the line (138B) connecting 1B and 1B is θ B Let's say.

[0082] By the quantities defined above, p 0B and point p on the projection line 130A B Distance between (135B) x pB is given by the following equation (14): x pB =d tanθ B =hB tanφ B tanθ B …(14) Conversely, the direction of the other party's information terminal 1A can be found using this equation (14).

[0083] According to the first embodiment, the information terminal 1B estimates the distance x d The estimated position p of the other party's information terminal 1A on the projection line 130A can be obtained. B teeth, x pB =x d …(15) This is the point where equation (15) holds.

[0084] By simultaneously solving equations (12), (14), and (15), the angle θ that gives the direction of the estimated position of the other party's information terminal 1A is obtained. B is calculated as follows: θ B =tan -1 {(x 0E -x A -x B )} / h B tanφ B} …(16) Or, φ B If it is difficult to measure, use the estimated value of g (Equation (9)) as an approximation of d, θ B =tan -1 {(x 0E -x A -x B )} / g} …(17) As shown in equation (17), θ B You may ask for:

[0085] In FIG. 11B, information terminal 1B is B The AR object mark 502 may be displayed superimposed on a point on the road in the direction of the arrow.

[0086] In addition, in FIG. 11C, information terminal 1B is BIf a candidate vehicle is identified, an AR object mark 504 may be displayed superimposed on the vehicle.

[0087] According to this embodiment, the vehicle to be met can be confirmed before it arrives. [Example]

[0088] In this embodiment, details of the meeting support on the vehicle side will be described.

[0089] As a hardware configuration on the vehicle side, as shown in Fig. 13A, a vehicle control device may have the functions of information terminal 1A, and the vehicle control device may display information to the user (driver). Alternatively, as shown in Fig. 13B, a configuration may be adopted in which vehicle control information and information used for display to the user are obtained from information terminal 1A. Note that the driver may also view the display of information terminal 1A.

[0090] When the vehicle control device controls the vehicle speed, it is performed in the manner described in the first embodiment.

[0091] The vehicle control device or information terminal 1A may present information to the vehicle occupants. For example, the displays shown in Figs. 14A, 14B, and 14C are as follows. Figs. 14A, 14B, and 14C are the same as those in the second embodiment, so details are omitted. Reference numeral 600 denotes a display screen of a display device, which displays the estimated distance of the passenger, etc. The state of the outside world may be displayed as a video image, or may be visible on a HUD (Head Up Display). Reference numerals 601, 603, and 605 denote explanatory messages superimposed on the screen. Reference numerals 602 and 604 denote AR object marks.

[0092] Furthermore, information about the passenger's appearance characteristics obtained from the server may also be used in estimating the position on the display screen.

[0093] Angle θ of the estimated position of the passenger as seen from the vehicle side AFrom the same consideration as in the second embodiment, is given by the following equation (18). θ A =tan -1 {(x 0E -x A -x B )} / h A tanφ A} …(18) where φ A is defined as shown in FIG. 15A.

[0094] FIG. 15A is a diagram including information terminal 1A, showing the positional relationship of each element on a vertical plane perpendicular to movement line 110B (a line when information terminal 1B is not moving, assuming that information terminal 1B moves parallel to movement line 110A of information terminal 1A). A projection line 130B is formed by vertically projecting movement line 110B onto ground surface 111. Information terminal 1A projects perpendicular line 131A from its own terminal to projection line 130B, and calculates angle 133A (φ A ) is measured. A Measurement is performed using the camera image and camera attitude information. Here, if the information terminal 1B, i.e., the passenger, cannot be identified, the projection line 130B is estimated by assuming a position on the side of the road where the passenger would normally stand. For example, it is acceptable to assume that the center line of the sidewalk on the side of the road is the projection line 130B, as this does not result in a large error.

[0095] Also, θ A is defined as shown in FIG. 15B. FIG. 15B is a diagram showing the positional relationship of each element projected vertically onto ground plane 111. In FIG. 15B, point 1A represents the point where the position of information terminal 1A is projected onto ground plane 111. p 0A (134A) is the foot of the perpendicular line drawn from 1A to the projection line 130B. And, on the ground 111 of FIG. 15B, 1A and p 0A and the estimated position p of the other party's information terminal 1B on the projection line 130B. A The angle (139A) of the line segment (138A) connecting 1A and 1B is θ A is.

[0096] Also, hA is the height of the information terminal 1A, more specifically, the height of the wireless signal transmission source of the information terminal 1A from the ground surface 111. A This information may be stored in the information terminal 1 A or may be held in the server 2 .

[0097] Or, θ A As in the case of the equation (17) in the second embodiment, the value calculated by the following equation (19) may be used. θ A =tan -1 {(x 0E -x A -x B )} / g} …(19) If there is no measurement error, θ A =θ B This becomes:

[0098] Furthermore, similarly to the second embodiment, when communication between the information terminals 1 is started or when it is estimated that the distance between them is equal to or less than a predetermined distance, the information terminal 1A may notify the user (driver) by voice or the like. By doing so, the user does not need to pay attention to the position of the passenger until the user is close to the passenger.

[0099] This embodiment allows passengers who are waiting to board to be confirmed before arrival. [Example]

[0100] In this embodiment, correction of the estimated value according to the situation at the meeting place will be described.

[0101] When there are many buildings and the reflected component of the wireless signal is large, the reception strength profile may be distorted from the function form of the above-mentioned equation (4). To take into account the effect of this distortion, in places with many buildings, the reception strength profile is set to the following equation (20) and the nearest position is estimated. P=k / {α(x-x0) 3 +(x-x0) 2 +g 2} …(20) In equation (20), the third-order term represents the effect of distortion.

[0102] Hereinafter, the function form of equation (4) will be referred to as L0 type, and the function form of equation (20) will be referred to as L1 type. Parameter fitting of L1 type is also performed for 1 / P. As in the first embodiment, when calculating using the least squares method, the measurement error of the received signal strength P is assumed to be constant, and the error of 1 / P is assumed to be P -2 It is also possible to perform weighted processing by assuming that the value is proportional to the value of the 1 / P=(1 / k) {α(x-x0) 3 +(x-x0) 2 +g 2} =A1x 3 +B1x 2 +C1x+D1…(21) Below, we estimate the closest position, i.e., x where the received signal strength is at its maximum, and the extreme point x where the received signal strength is at its maximum is the extreme point x where the received signal strength is at its minimum in equation (21). The first-order differential of equation (21) is given by the following equation (22), so we can see that the first-order differential value of equation (21) is 0 at x0. (1 / P)´=(1 / k) {3α(x-x0) 2 +2(x-x0)} …(22) Therefore, among the estimated values of x0 below, the x0 for which the second derivative of equation (21) is positive is the x for which the received signal strength is at its maximum.

[0103] Now, equation (21) is linear with respect to the parameters A1, B1, C1, and D1, and is found from the reception intensity data by the least squares method. The relationship between the parameters A1, B1, C1, and D1 and the parameters in equation (21) is as follows: α / k=A1…(23) (1-3αx0) / k=B1…(24) -(2x0-3αx0 2 ) / k=C1…(25) (x0 2 +g 2 -αx0 3 ) / k=D1…(26) By solving equations (23), (24), and (25) simultaneously, two x0 values are obtained as shown below. x0={-B1±(B12 -3A1C1) 1 / 2} / (3A1) …(27) On the other hand, the second derivative of (21) is given by the following equation (28). (1 / P)´´=6A1x+2B1…(28) The second-order differential value expressed by this inner equation (28) is positive as follows: x0={-B1+(B1 2 -3A1C1) 1 / 2} / (3A1) …(29) where: J=B1 2 -3A1C1…(30) When J>0, there is an extreme value at x0. When J≦0, this means that equation (21) does not have an extreme value, so L1-type fitting cannot be used.

[0104] Assuming that x0 is the value of equation (29), the values of the other parameters α and g are shown in equations (31) and (32). α=(1 / 3)(2x0-C1 / B1) / (x0 2 -(C1 / B1)x0) …(31) g=(αx0 3 -x0 2 -αD1 / A1) …(32) Hereafter, we set J>0 and use x0 in equation (29) as the estimated value of x that gives the maximum received signal strength. 0E The processing and control in other embodiments are carried out as follows.

[0105] In addition, since the L1 type approximation becomes poor in the range of x where the absolute value of x-x0 is large, the reception strength data used in fitting using the L1 type may be limited to that close to the estimated x0.

[0106] Next, a method for evaluating the amount of building wall surface that contributes to the reflection of radio waves (hereinafter referred to as the amount of building) will be described.

[0107] FIG. 16 is an explanatory diagram illustrating a method for evaluating the influence of buildings that contribute to radio wave reflection in this embodiment. The influence of buildings is evaluated using, for example, map information. FIG. 16 shows map information showing the status of buildings 4 around meeting place 3. The map information may be obtained from server 2 or may be stored on the device itself. In FIG. 16, buildings 4A to 4F are surrounding meeting place 3. Since the influence of buildings 4 is problematic due to the reflection of radio signals, the influence is evaluated based on the length of the walls facing the vehicle's route within a predetermined range 5 from meeting place 3 along the route of vehicle 1A.

[0108] The predetermined range 5 may be set wider for places of high importance. For example, as the car approaches, it may be divided into areas 5A, 5B, 5C, and 5D depending on whether the area is in front of or ahead of the meeting place 3, and whether it is along the road on the side of the meeting place 3 or on the opposite side. Since passenger position estimation is performed with car 1A in front, buildings 4 closer to the meeting place 3 are important, and in terms of distance, buildings 4 along the road on the side of the meeting place 3 are also important.

[0109] The wall lengths 6 facing the vehicle route within the set predetermined range 5 are 6A, 6B, 6C, 6D, and 6E in the case of Figure 16. The sum of these wall lengths is the amount of building wall surface that contributes to the reflection of radio waves.

[0110] One control method is to use the L0 type as the fitting function form when the amount of wall surface of the building 4 is less than a predetermined amount, depending on the condition of the building 4, as shown in Figure 17A, and to use the L1 type when it exceeds the predetermined amount. Alternatively, as shown in Figure 17B, since the approximation of the L1 type becomes poor in the range of x where the absolute value of x-x0 is large, it is also possible to use the L0 type when the distance to the meeting place 3 is greater than a predetermined value, and the L1 type when it is less than the predetermined value. Note that when switching fitting functions depending on the position of the car, the estimated distance to the passenger may change significantly when switching, so a position-weighted average of the estimated values obtained by the two fitting functions for a certain section may be used.

[0111] Alternatively, the above two methods may be combined. For example, the amount of wall surface of the building M B The fitting function is switched by the predetermined distance T P may be changed. T p =f(M B ) …(33) Since the greater the amount of wall surface of a building, the greater the possibility that the distortion of the reception intensity profile will increase, f is set to a monotonically increasing function in the broad sense. Alternatively, threshold control as shown in FIG. 17C may be used. T P1 <T P2 …(34) Set it so that:

[0112] When using an L1 type fitting function, if the value of J (Equation (30)) is 0 or negative, this means that the fitted function does not have an extreme value. Therefore, regardless of the above conditions, the estimated value using the L0 type fitting function will be used.

[0113] According to this embodiment, the estimated value can be corrected depending on the situation at the meeting place. [Example]

[0114] In this embodiment, when the person to be met can be identified before the person arrives, the distance to the arrival is estimated based on information about the direction in which the person can be seen.

[0115] In this embodiment, the case where the vehicle-side information terminal 1A identifies the passengers will be described, but the reverse case is also applicable. The derivation of the relationships between the various quantities is the same as in the third embodiment. The symbols are the same as those in Figs. 15A and 15B.

[0116] First, the passenger direction θ identified by the information terminal 1A A Measure the angle θ A The measurement is performed using the camera image and camera posture information. Aand the distance x to the closest position d The relationship is the same as that considered in the third embodiment, and is given by the following equation (35) from equation (18). x d =x 0E -x A -x B =h A tanφ A tanθ A …(35) This gives an estimate of the closest travel distance x0 based on the direction measurement. 0E is obtained.

[0117] In the first embodiment, x is calculated from the change in the received radio wave strength. 0E However, the x 0E Instead of x obtained by equation (35), 0E The vehicle speed is controlled using the above method. In order to prevent shocks in the vehicle speed control when switching from control based on the radio wave reception strength to control based on camera measurements, the switching period is determined by using both methods together and the x estimated by each method. 0E Alternatively, a weighted average of the values of 1 to 3 may be used to gradually switch the values.

[0118] Furthermore, the car control device adjusts the stopping position target by x so that the car door is in front of the passenger at the stopping position. 0E Therefore, the position may be shifted by the difference in the position of the information terminal 1A and the car door in the traveling direction.

[0119] In addition, if the passenger can identify the location of the car, the information terminal 1B will display x 0E may be estimated and the information transmitted to the vehicle, and the vehicle's control device may use the value to control the vehicle.

[0120] When the passenger side performs the camera measurement, the following relational expression (36) is used in accordance with the second embodiment in correspondence with the expression (16). x d =x 0E -x A -x B =h B tanφ B tanθ B…(36) Here, the angle θ B The measurement is performed by the information terminal 1B using the camera image and the camera attitude information.

[0121] As a result, more precise control of the stopping position can be achieved. [Example]

[0122] In this embodiment, a description will be given of how to deal with the case where the bus fails to stop properly in front of the passenger.

[0123] If the vehicle is unable to stop in front of the passenger, the vehicle that is closest to the passenger when the reception strength reaches its maximum is the vehicle they are waiting for, so that vehicle is photographed as shown in Fig. 18A and the photographed image is used for subsequent guidance. The image photographed from the front position is displayed on the screen of information terminal 1B as shown in Fig. 18B, and if the vehicle can be tracked by the camera, a mark is superimposed on the image of the target vehicle.

[0124] Once the vehicle's control system detects that the passenger has passed, it slows down the vehicle as far as is safe and brings the vehicle to a stop.

[0125] As a result, even if the vehicle cannot stop directly in front of the passenger, the passenger can be guided to the vehicle. [Example]

[0126] In this embodiment, a modified example of the user interface will be described.

[0127] In this embodiment, when the smartphone serving as the passenger's information terminal 1C that called the taxi displays the taxi's location on a map, the mark displaying the location of the taxi, i.e., the other party's information terminal 1A, changes depending on whether or not a direct communication wireless signal is being received. Here, the direct communication wireless signal refers to a wireless signal such as a beacon signal accompanied by the aforementioned identification signal, and may be referred to simply as a wireless signal in this embodiment. It is also assumed that the other party's information terminal 1A transmits a wireless signal, and the information terminal 1C receives the wireless signal.

[0128] FIG. 19A shows the display screen of information terminal 1C in this embodiment when the location of the other party's information terminal 1A is still far from the meeting place and information terminal 1C is not receiving a wireless signal. At this time, information terminal 1C displays a mark drawn with a dotted line as mark 1000 representing the other party's information terminal 1A on the map. On the other hand, FIG. 19B shows the display screen of information terminal 1C in this embodiment when the other party's information terminal 1A is approaching the meeting place and information terminal 1C is receiving a wireless signal. At this time, information terminal 1C displays a mark drawn with a solid line as mark 1001 representing the other party's information terminal 1A on the map. Note that, although the above example uses dotted and solid lines to differentiate between mark 1000 and mark 1001, this is not limiting. For example, any display difference may be used, such as blurring, changing the color, or enclosing them in a frame. Note that in FIGS. 19A and 19B, mark 1010 represents the location of the own information terminal.

[0129] As for how the mark is displayed, once the other party's information terminal 1A approaches the meeting place and the information terminal 1C starts receiving a wireless signal, the mark 1001 may continue to be displayed, or if for some reason the information terminal 1C is no longer able to receive a wireless signal, the mark may return to being displayed as mark 1000.

[0130] Before information terminal 1C is able to receive wireless signals, information terminal 1C acquires display position information of counterpart information terminal 1A on a map from counterpart information terminal 1A by communication other than direct communication, such as via the Internet. The counterpart information terminal 1A measures its own location using GPS or the like. After information terminal 1C is able to receive wireless signals, information terminal 1C may determine the display position of counterpart information terminal 1A on a map by a method of estimating it from changes in the reception strength of the wireless signal, as described above. Note that, to prevent a sudden change in the display position of counterpart information terminal 1A before and after starting to receive wireless signals, information terminal 1C may gradually change the display position of counterpart information terminal 1A from the position acquired from counterpart information terminal 1A to the position determined by a method of estimating it from changes in the reception strength of the wireless signal.

[0131] Furthermore, the information terminal 1C may change the mark representing the other information terminal 1A in stages depending not only on whether or not a wireless signal is received, but also on the strength of the wireless signal reception. For example, as shown in Fig. 20, the information terminal 1C displays the marks (a) to (d) by lengthening the length of the bar as the reception strength increases.

[0132] Furthermore, if information terminal 1A is unable to measure its own location due to reasons such as inability to receive GPS radio waves, and is also unable to estimate its location using direct communication radio signals, information terminal 1C may display an explanatory message 1005 indicating that its location is unknown, as shown in Figure 21, instead of displaying a mark indicating the location of information terminal 1A on a map.

[0133] Furthermore, information terminal 1C not only displays the position of the other information terminal 1A, but also, when the other information terminal 1A approaches, can switch to a service that allows the user to view the other information terminal 1A through a camera image or by looking directly at it, as shown in Figures 11A to 11C. This service switching may be performed automatically by information terminal 1C, or a user instruction may be received via input box 1007 or the like shown in Figure 22. Furthermore, information terminal 1C and information terminal 1B may cooperate to start a new service on another information terminal 1B (for example, an HMD).

[0134] If the information terminal owned by the user is an information terminal 1D such as a watch, notification corresponding to the reception strength of the wireless signal from the other information terminal 1A may be made by voice or vibration instead of by displaying a mark. That is, in the case of voice notification, as shown in Fig. 23, the information terminal 1D notifies the other information terminal 1A (taxi) that it is approaching or notifies the change in reception strength when it receives a wireless signal for direct communication. In addition, in the case of vibration notification, as shown in Fig. 24, the information terminal 1D generates vibrations in a preset vibration pattern when the reception strength changes.

[0135] In the above description, the other party's information terminal 1A transmits a wireless signal and the information terminal 1C receives the wireless signal, but the information terminal 1C may transmit a wireless signal and the information terminal 1A may receive the wireless signal. In this case, the information terminal 1A obtains the reception strength information of the wireless signal from the information terminal 1C by direct communication or other communication means.

[0136] Furthermore, the type of information terminal is not limited to the example. For example, the information terminal in the taxi may be the local information terminal 1A, and the customer's smartphone may be the other party's information terminal 1C, and the information terminal 1A may display a mark according to the reception strength, or may issue a voice notification or a vibration notification. An example of the display on the information terminal 1A is shown in FIG. 25. Mark 1200 indicates the position of the local information terminal 1A, and mark 1210 indicates the position of the other party's information terminal 1C. The information terminal 1A then changes the shape of the mark 1210 according to the reception strength of the wireless signal. Specific variations are basically the same as those described above, and therefore detailed explanations will be omitted.

[0137] As described above, if the user knows whether or not a wireless signal is being received and the reception strength, the user can visually grasp the distance from the user's own information terminal to the other information terminal 1A for which he or she is waiting. Also, if the user knows that a wireless signal can be received and that the other information terminal 1A has come close, it is suitable for the user to grasp the timing to switch services, such as switching to the meeting support display described in other embodiments. Furthermore, the information terminal 1C can also control the content of meeting support according to the location of the other information terminal 1A in a way that is convenient for the user, improving user convenience. [Example]

[0138] In this embodiment, a modified example of a meeting support service using camera images will be described.

[0139] Information terminal 1C acquires an image (a still image or a video, hereinafter referred to as a "other information terminal image") of the direction in which information terminal 1C is located, taken by other information terminal 1A based on the estimated position of information terminal 1C, and identifies the position of information terminal 1C in the image. Information terminal 1C then transmits the identified position information of its own information terminal 1C in the other information terminal image to other information terminal 1A, and other information terminal 1A identifies the position of information terminal 1C from the received position information. Here, the position in the image means the position in the image drawing area.

[0140] A detailed procedure will be described. First, the information terminal 1C displays a counterpart information terminal image 2000 as shown in FIG. 26A. Note that, as shown in FIG. 26B, the counterpart information terminal image 2000 may be displayed superimposed on the map display screen in the seventh embodiment. In this example, the counterpart information terminal image 2000 shows a person 2001a who is the user carrying the information terminal 1C as seen from the information terminal 1A, and a person 2001b who is not the user. The information terminal 1C accepts a user input that identifies a person in the image who is the user or a meeting location. The user input that identifies a person or a meeting location is, for example, an input in which a position on the display screen is specified by touching with the user's finger, and the person or the meeting location is identified by the specified position. Note that the input method using a touch operation is one example, and any method that can specify a person or a location is acceptable, and is not limited to a touch operation. The same applies to the following input method for specifying a person or a location. Furthermore, an input that identifies a person may specify the location where the person is located as the meeting location, or an input that specifies the meeting location may specify the person present there. Alternatively, the information terminal 1C may identify the user person 2001a by image analysis, without relying on the user's input to identify the person. When analyzing the image, the information terminal 1C may use an image of the user taken by the information terminal 1C at the time of the meeting for personal identification.

[0141] After identifying the person to be met (i.e., the user himself / herself) or the meeting location, information terminal 1C displays a person identification mark 2010 or a meeting location identification mark 2011 superimposed on the other party's information terminal image 2000. Information terminal 1C may display both marks. After identifying the person or the meeting location, information terminal 1C transmits position information of the identified person in the other party's information terminal image 2000 to other party's information terminal 1A. The other party's information terminal 1A recognizes the person to be met and the meeting location based on the position information in the image acquired from information terminal 1C.

[0142] A variation of the person identification method may be a method as shown in FIG. 27. That is, the information terminal 1C displays images (2021a, 2021b) of the person portion extracted from the other party's information terminal image and accepts the user's input action for person identification. In this example, the image 2021a is the person image corresponding to the user, and after accepting the user's action for identifying the image 2021a, the information terminal 1C displays, for example, an identification mark 2020 indicating that the image has been identified. After accepting the user's action for identifying the person image, the information terminal 1C notifies the other party's information terminal 1A of the identification result. The other party's information terminal 1A recognizes who the person is at the meeting based on the person image identification result notified from the information terminal 1C. The extraction of the images (2021a, 2021b) may be performed by the other party's information terminal 1A, the information terminal 1C, or even the server 2.

[0143] Next, a method for the user carrying information terminal 1C to confirm the recognition result of the person to be met and the meeting location by the other party's information terminal 1A will be described.

[0144] The other party's information terminal 1A recognizes the person to be met and the meeting location by estimating the location based on the reception strength of the wireless signal of direct communication or by the location information acquired from information terminal 1C. Information terminal 1C acquires information on the recognition result by the other party's information terminal 1A and displays the recognition result as shown in FIG. 28. In the display example of FIG. 28, information terminal 1C displays a specific mark 2030 indicating the person to be met recognized by the other party's information terminal and a specific mark 2031 indicating the meeting location superimposed on a other party's information terminal image 2100. The superimposition of these marks may be performed by information terminal 1C, or the other party's information terminal image 2100 may be acquired by information terminal 1A as an image on which the marks are superimposed.

[0145] The user of information terminal 1C can judge whether the recognition result is correct by looking at specific mark 2030 of the person to be met recognized by other party information terminal 1A and specific mark 2031 of the meeting location. After presenting other party information terminal image 2100 to the user, information terminal 1C accepts correction input from the user.

[0146] If corrections are necessary, the user inputs the corrections. Correction input is the same as in the case of FIG. 26A, where the user inputs the user's own position or the meeting location in the image by touching or other means. FIG. 28 shows an example of a case where the meeting target is person 2001a, but the other party's information terminal 1A erroneously recognizes the meeting target as person 2001b. After receiving correction input from the user, such as that the meeting target is person 2001a (i.e., the user himself / herself), information terminal 1C notifies the other party's information terminal 1A of the correction information. Information terminal 1A corrects the meeting target person and the meeting location using the correction information notified by information terminal 1C.

[0147] The person to be met may be corrected in a display format that lists images of people as shown in Fig. 29. In the list display of person images, a method such as superimposing a specific mark 2040 may be used to display the image of the person that the other party's information terminal 1A recognizes as the person to be met, so that the user of the information terminal 1C can determine whether the recognition result is correct. When correcting, the information terminal 1C accepts input of the correct person image, as in Fig. 27.

[0148] The type of information terminal is not limited to the example. For example, the taxi's information terminal may be the local information terminal 1A, and the customer's HMD may be the other party's information terminal 1B, and the information terminal 1A may determine whether the customer information terminal 1B correctly recognized the target vehicle for the meeting. An example of a display on the information terminal 1A is shown in FIG. 30. The information terminal 1A displays the other party's information terminal image 2200, which shows cars 2051a and 2051b. A specific mark 2060 indicating that the information terminal 1B has recognized the target vehicle for the meeting is also superimposed and displayed. The vehicle corresponding to the information terminal 1A is 2051a, but this example shows a state in which the information terminal 1B has mistakenly recognized a different vehicle, 2051b, as the target vehicle for the meeting. To correct the incorrect recognition of the target vehicle for the meeting, the information terminal 1A may make a correction determination using its own image analysis or other means, or may accept a correction input from the user, as described above. Since the user of information terminal 1A may be driving a car, the user may input corrections not by touching the screen or other operations, but by speaking the identification symbols (2052a, 2052b) for identifying the car that have been added and superimposed by information terminal 1A. Other embodiments are basically the same as those described above, and therefore detailed descriptions thereof will be omitted.

[0149] Figure 31 is a control flow diagram for meeting support in this embodiment. Figure 31 is based on the flow diagram of Figure 9, and the processing from steps S51A and S51B onwards to steps S54A and S54B is different, so those parts will be explained below. The other steps are the same as those in Figure 9, so explanations will be omitted.

[0150] In steps S51A and S51B, each information terminal 1 provides the user or the vehicle control device with information to support the meeting. In addition to displaying the support information on its own information terminal, the information displayed on its own information terminal is transmitted to the other information terminal.

[0151] In steps S52A and S52B, if the user inputs correction information such as the vehicle or person to be met, the input is accepted.

[0152] In steps S53A and S53B, if there is any correction information input by the user or determined by the information terminal, that information is transmitted to the other information terminal.

[0153] In steps S54A and S53B, if there is any correction information, the support information is corrected.

[0154] As a result, by acquiring an image of the other party's information terminal, it is possible to confirm that the other party's information terminal recognizes the location of its own information terminal and to notify the other party's information terminal of the location of its own information terminal, thereby enabling more accurate support for rendezvous. [Example]

[0155] In this embodiment, a modified example will be described in which the information terminal 1 can be easily identified in an image.

[0156] In the above-described embodiments, when the information terminal 1 determines a person or a vehicle in an image as a meeting target, the information terminal 1 either accepts a user's input of the determination or uses a determination based on image analysis. In this embodiment, instead of these methods, the information terminal 1 identifies the person or vehicle using identification information transmitted as an optical signal using the light-emitting device 3000.

[0157] Fig. 32 shows a display screen 500 of the information terminal 1B, which is an HMD in this embodiment. In the case of a see-through HMD, the user can see the outside world directly. In the case of a video see-through HMD, the user can see the outside world as a video image. In the case of a see-through HMD, it is assumed that the outside world is also captured by a camera.

[0158] FIG. 32 shows a situation in which the user faces a direction in which the vehicle to be met is estimated to be located based on the reception strength of the wireless signal of direct communication. In this example, there are two candidate vehicles (3001a, 3001b) within the user's field of view. Light-emitting devices 3000a and 3000b are installed on the vehicle bodies, respectively. In the case of information terminal 1A, when information terminal 1A is mounted in a vehicle, light-emitting device 3000 is assumed to be separate and installed in a position visible from the outside. In this situation, information terminal 1A transmits its own identification information as an optical signal using light-emitting device 3000. This identification information may be the same as the identification information of the wireless signal of direct communication, but information terminal 1A notifies partner information terminal 1B in advance. Alternatively, information terminal 1B may specify the identification information.

[0159] The transmission of identification information by optical signals using the light-emitting device 3000 is performed by modulating the light emission intensity, as shown in FIG. 33, for example. FIG. 33 shows communication using a bit string in which light emission corresponds to "1" and no light emission corresponds to "0," but this is not limiting. For example, multi-value modulation may be used. To ensure that the camera can capture the modulation of the light emission intensity reliably, the modulation speed of the light emission intensity is set slower than the frame rate of the camera on the capturing side. For example, when the camera frame rate is 60 frames per second, the modulation speed is set to about 20 baud.

[0160] Information terminal 1B captures video with a camera and extracts the areas of light-emitting devices 3000a and 3000b through image analysis. Information terminal 1B then restores the information transmitted from each vehicle (3001a, 3001b) from changes in brightness in that area. Information terminal 1B then verifies the identification information of information terminal 1A, the vehicle to be met, either from a notification it has received in advance or from a designation it has made. This information identifies the vehicle to be met. In FIG. 33, the vehicle to be met is 3001a, and information terminal 1B displays a superimposed identification mark 3050 using an AR object indicating that vehicle 3001a is the vehicle to be met. If, by chance, there is a target transmitting the same identification information as the identification information of information terminal 1A, the vehicle to be met, information terminal 1B may request information terminal 1A to change the identification information to be transmitted.

[0161] Note that the identification information may be transmitted from both information terminals 1, and each may confirm the other information terminal 1. The combination of types of information terminals 1 is not limited to the above. The information terminal 1 that transmits information from the light-emitting device 3000 using an optical signal may receive an image of the other information terminal 1 and confirm the information transmitted by its own light-emitting device 3000. Furthermore, the server 2 may restore the information transmitted from the light-emitting device 3000 in the image, and the server 2 may notify the information terminal 1 of which person, car, or information terminal 1 is the person, car, or information terminal 1 that is the meeting target in the image.

[0162] The method of this embodiment may be combined with other embodiments as a means for identifying the person, car, or information terminal 1 to be met.

[0163] Figure 34 is a control flow diagram for meeting support in this embodiment. Based on the flow diagram in Figure 9, the above-mentioned processing is added as steps S71A and S71B. The other steps are the same as those in Figure 9, so their explanation will be omitted.

[0164] In steps S71A and S71B, one information terminal 1 uses the light-emitting device 3000 to transmit its own identification information to the other information terminal 1. The information terminal 1 that receives the transmitted identification information captures a video in the direction in which it presumes the transmitting information terminal 1 is located. Here, both information terminals 1 may be the senders of the identification information. This video is analyzed by the information terminal 1 that receives the identification information, the information terminal 1 that transmits the identification information, or the server 2, and the transmitting information terminal 1 in the image is identified.

[0165] As described above, by transmitting identification information using a light-emitting device by information terminal 1 and analyzing a video captured of information terminal 1 transmitting the identification information, the exact location of the other information terminal 1 can be determined, thereby improving the reliability of the meeting support system.

[0166] Although the above embodiments have been described in detail to clearly explain the present invention, the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. Furthermore, the functions described above may be implemented in part or in whole by hardware, or by software program processing. The programs and data constituting the functions may be stored in a computer-readable storage medium or in a device on a communication network. [Explanation of symbols]

[0167] 1, 1A to 1D: information terminal, 2: server, 3: meeting place, 4: building, 5: predetermined range, 6: wall length, 9: communication network, 10: housing, 11: display surface, 101: processor, 102: memory, 103: display device, 107: operation input unit, 110A, 110B: movement line, 111: ground surface, 112A, 112B: starting point, 113, 115A, 115B: distance, 114A, 114B: movement distance, 120: direct wave path, 130A, 130B: projection line, 500, 600: display screen, 501, 503, 505, 601, 603, 605, 1005: explanatory message ,502, 504, 602, 604: AR object marks, 1000, 1001, 1010, 1200, 1210: marks, 1007: input box, 1100: vibration generator, 2000, 2100, 2200: image of partner information terminal, 2001a, 2001b: person, 2010, 2011, 2020, 2030, 2031, 2040, 2060, 3050: specific marks, 2021a, 2021b: images, 2051a, 2051b, 3001a, 3001b: cars, 2052a, 2052b: identification symbols, 3000, 3000a, 3000b: light-emitting devices

Claims

1. An information terminal, a first communication unit that receives information about the location of an external terminal via a communication network; a second communication unit that directly receives a wireless signal including identification information from the external terminal; A display unit; a control unit, The control unit displaying a first mark corresponding to the external terminal on a map displayed on the display unit based on information relating to the location of the external terminal received by the first communication unit via a communication network; When at least one of the information terminal and the external terminal moves, the information terminal becomes able to directly receive the wireless signal from the external terminal, the information terminal measures the reception strength of the wireless signal directly received from the external terminal, a control mode for estimating a relative positional relationship between the information terminal and the external terminal based on the measured reception strength of the wireless signal, and controlling the display unit to display a second mark corresponding to the external terminal and different from the first mark; An information terminal, characterized in that the form of the second mark displayed on the display unit is changed according to the strength of the reception strength.

2. 2. The information terminal according to claim 1, The information terminal is characterized in that, after the second mark is displayed on the display unit, when the information terminal is no longer able to directly receive the wireless signal from the external terminal, the control unit controls the mark displayed on the display unit to return to the first mark.

3. 2. The information terminal according to claim 1, The information terminal is characterized in that the control unit controls the display unit to display the estimated relative positional relationship between the information terminal and the external terminal.

4. 4. The information terminal according to claim 3, The control unit controls the display unit to display information regarding the distance from the information terminal to the external terminal as the estimated relative positional relationship between the information terminal and the external terminal.

5. 4. The information terminal according to claim 3, The information terminal is characterized in that the control unit controls the display unit to display an arrow indicating the direction of the position of the external terminal as the estimated relative positional relationship between the information terminal and the external terminal.

6. 4. The information terminal according to claim 3, The control unit controls the display unit to display a message indicating the positional relationship with the external terminal as the estimated relative positional relationship between the information terminal and the external terminal.

7. 7. An information terminal according to claim 1, The information terminal is characterized in that, in the control mode, after the second mark is displayed on the display unit, when the information terminal and the external terminal become further closer to each other, the control unit automatically switches the display on the display unit to a display different from the second mark.

8. 7. An information terminal according to claim 1, Furthermore, it is equipped with a camera, The information terminal is characterized in that, when the control unit receives a user instruction in the control mode, it controls the display unit to display an image captured by the camera instead of the second mark.

9. 9. The information terminal according to claim 8, The control unit controls the display unit to superimpose an AR object mark for identifying the external terminal on the captured image when the external terminal is included in the image captured by the camera.

10. 7. An information terminal according to claim 1, The information terminal is characterized in that the control unit controls the display of the second mark to be displayed on a map.

11. A system including a first information terminal, a second information terminal, and a server, the first information terminal includes a transmitter that transmits a wireless signal accompanied by identification information; the second information terminal includes a first communication unit that receives information related to the location of the first information terminal via the server, a second communication unit that receives the wireless signal accompanied by identification information from the first information terminal without going through the server, a display unit, and a control unit; The second information terminal receiving information about a location of the first information terminal by the first communication unit via the server, and displaying a first mark corresponding to the first information terminal on a map displayed on the display unit based on the received information about the location of the first information terminal; When at least one of the first information terminal and the second information terminal moves and the wireless signal transmitted by the transmitting unit of the first information terminal becomes receivable by the second communication unit without going through the server, the receiving unit measures the reception strength of the wireless signal received from the first information terminal without going through the server; estimating a relative positional relationship between the first information terminal and the second information terminal based on the measured reception strength of the wireless signal, and controlling the display unit to display a second mark corresponding to the first information terminal and different from the first mark; A system characterized in that the form of the second mark displayed on the display unit of the second information terminal is changed depending on the strength of the reception strength at the second information terminal of the wireless signal transmitted by the transmitter of the first information terminal.

12. 12. The system of claim 11, The system is characterized in that, after the second information terminal displays the second mark on the display unit, when the second information terminal is no longer able to directly receive the wireless signal from the first information terminal, the second information terminal controls the mark displayed on the display unit to return to the first mark.

13. 12. The system of claim 11, The system is characterized in that the second information terminal controls the display unit to display the estimated relative positional relationship between the second information terminal and the first information terminal.

14. 14. The system of claim 13, The system is characterized in that the second information terminal controls the display unit to display information regarding the distance from the second information terminal to the first information terminal as the estimated relative positional relationship between the second information terminal and the first information terminal.

15. 14. The system of claim 13, The system is characterized in that the second information terminal controls the display unit to display an arrow indicating the direction of the position of the first information terminal as the estimated relative positional relationship between the second information terminal and the first information terminal.

16. 14. The system of claim 13, The system is characterized in that the second information terminal controls the display unit to display a message indicating the positional relationship with the first information terminal as the estimated relative positional relationship between the second information terminal and the first information terminal.

17. 17. A system according to any one of claims 11 to 16, comprising: The system is characterized in that, after the second mark is displayed on the display unit, when the second information terminal and the first information terminal come closer to each other, the second information terminal automatically switches the display on the display unit to a display different from the second mark.

18. 17. A system according to any one of claims 11 to 16, comprising: the second information terminal further includes a camera; A system characterized in that, when the second information terminal receives a user instruction, it controls the display unit to display an image captured by the camera instead of the second mark.

19. 20. The system of claim 18, The system is characterized in that, when the first information terminal is included in an image captured by the camera, the second information terminal controls the display unit to superimpose an AR object mark for identifying the first information terminal on the captured image.

20. 17. A system according to any one of claims 11 to 16, comprising: The system is characterized in that the second information terminal controls the display of the second mark so as to be displayed on a map.

Images