Information terminal and guidance method using information terminal

Abstract

This information terminal generates an external video in which an external subject appears, calculates a value in a terminal-world coordinate system of the information terminal, the value being of a direction vector from a terminal position of the information terminal toward each of a plurality of external feature points that are a part of the external subject, acquires each feature amount of a plurality of external feature points in a marker video in which the external subject is photographed in advance and a coordinate value, in an in-area-world coordinate system, of each of the plurality of external feature points, and compares the feature amount of each of the plurality of external feature points in the external video to the feature amount of each of the plurality of external feature points in the marker video, and on the basis of the comparison result, the direction vector value in the terminal-world coordinate system of the information terminal, said direction vector being directed from the terminal position of the information terminal toward each of the plurality of external feature points appearing in the external video, and the coordinate values in the in-area-world coordinate system of each of the plurality of external feature points appearing in the marker video in which the external subject was photographed in advance, the information terminal calculates a coordinate conversion parameter between the terminal-world coordinate system of the information terminal and the in-area-world coordinate system, calculates the terminal position of the information terminal in the in-area-world coordinate system on the basis of the coordinate conversion parameter, and guides the information terminal to a target location defined by the in-area-world coordinate system.

Description

Information terminal and guidance method using information terminal

[0001] The present invention relates to an information terminal and a guidance method using the information terminal.

[0002] As a technology for guiding a user to a destination, the information transmission / reception device in Patent Document 1 is described as comprising "a current location acquisition means provided on the side of a first user who is to meet a second user at a predetermined meeting place, and which acquires the current location of the first user, a meeting place acquisition means for acquiring the meeting place for the first user and the second user, a traveling direction prediction means for predicting the traveling direction of the first user who will arrive at the meeting place based on the positional relationship between the current location of the first user and the meeting place, an imaging command generation means for generating an imaging command for capturing an image of the meeting place in a predetermined imaging direction based on the traveling direction predicted by the traveling direction prediction means, and a transmission / reception means for sending the generated imaging command to the second user (abstract excerpt)."

[0003] Japanese Patent Application Laid-Open No. 2005-17074

[0004] In the information transmission and reception device of Patent Document 1, a user heading to a meeting place by car can tell that they have arrived near the meeting place by comparing the image taken by the other user at the meeting place with the scenery in front of the car, but there is a problem in that they cannot know the exact location of the other user.

[0005] The present invention has been made in view of the above-mentioned problems, and relates to an information terminal that accurately notifies a user of the location of a target location such as a meeting place, and a guidance method using the information terminal.

[0006] In order to solve the above-mentioned problems, the present invention has the configuration described in the claims. For example, an information terminal includes a processor and a camera, the camera generates an external world image in which an external world subject is reflected, the processor acquires feature amounts of each of the external world feature points reflected in a landmark image previously captured of the external world subject and coordinate values ​​of each of the external world feature points in an in-area world coordinate system, and compares the feature amounts of each of the external world feature points reflected in the external world image with the feature amounts of each of the external world feature points reflected in the landmark image. and a coordinate transformation parameter between the terminal world coordinate system of the information terminal and the intra-area world coordinate system based on the result of the calculation, values ​​in the terminal world coordinate system of the information terminal of direction vector values ​​from the terminal position of the information terminal toward each of the plurality of external feature points reflected in the external world image, and coordinate values ​​in the intra-area world coordinate system of the plurality of external feature points reflected in a landmark image of the external subject captured in advance, and outputs guidance information for guiding the information terminal to a target point defined in the intra-area world coordinate system.

[0007] According to the present invention, it is possible to provide an information terminal and a guidance method using the information terminal that can accurately notify a user of the location of a target location such as a meeting place. Note that problems, configurations, and effects other than those described above will become clear from the description of the following embodiments.

[0008] 1 is a configuration diagram of a meeting support system according to an embodiment of the present invention. FIG. 2 is a hardware configuration diagram of an information terminal. FIG. 3 is a hardware configuration diagram of a server. FIG. 4 is a diagram illustrating an example of usage of the meeting support system according to an embodiment of the present invention. FIG. 5 is a diagram illustrating an image captured from a user information terminal. FIG. 6 is a diagram illustrating an image captured from an other person's information terminal. FIG. 7 is a diagram illustrating an example of external world feature point information. FIG. 8 is a diagram illustrating an example of a correspondence relationship between external world feature points. FIG. 9 is a diagram illustrating an example of measuring coordinate values ​​of external world feature points in a terminal world coordinate system. FIG. 10 is a diagram illustrating an example of evaluating feature quantities of external world feature points. FIG. 11 is a diagram illustrating the contents of a method (1) for acquiring an inter-terminal positional relationship between an other person's information terminal and a user information terminal, and is an explanatory diagram of two directional vectors defined in real space. FIG. 12 is a diagram illustrating the contents of a method (1) for acquiring an inter-terminal positional relationship between an other person's information terminal and a user information terminal, and is a diagram illustrating the relationship between the numerical expressions of two directional vectors in the terminal world coordinate systems of the user information terminal and the other person's information terminal. FIG. 13 is a diagram illustrating a concept of separating the positional relationship between an external world feature point and an HMD 1B, which is the other person's information terminal, into a horizontal projection plane and a vertical projection plane. 13A and 13B are diagrams showing the contents of a method (2) for acquiring the inter-terminal positional relationship of an other information terminal with respect to a user information terminal, and showing a method for calculating coordinate values ​​of the other information terminal in the coordinate system of the user information terminal in a projection plane. FIG. 13B are diagrams showing the contents of a method (2) for acquiring the inter-terminal positional relationship of an other information terminal with respect to a user information terminal, and showing a method for calculating coordinate values ​​of the other information terminal in the coordinate system of the user information terminal in three-dimensional space. FIG. 13C are diagrams showing examples of processing for special situations in the processing of FIG. 13A. FIG. 13C are diagrams showing the contents of a method (2) for acquiring the inter-terminal positional relationship of an other information terminal with respect to a user information terminal, and showing an explanatory diagram of two directional vectors defined in real space. FIG. 13D are diagrams showing an example of changing to a low-distortion shooting mode when shooting an outside world image. FIG. 13D are diagrams showing an example of changing to a wide-angle shooting mode when shooting an outside world image. FIG. 13E are diagrams showing an example of displaying an image shot by a partner terminal when shooting an outside world image. FIG. 13F are diagrams showing an example of displaying a virtual object at a shooting target point when shooting an outside world image. FIG. 13F are diagrams showing an example of displaying a prompt to take additional photos when shooting an outside world image. 10A and 10B are diagrams illustrating an example in which, when capturing an external image, the partner terminal displays a message indicating that additional capture is not possible.10 is a diagram showing an example of information display on a user information terminal (vehicle-mounted terminal 1A) based on terminal position estimation for meeting support. FIG. 11 is a diagram showing an example of information display on an other person's information terminal (HMD 1B) based on terminal position estimation for meeting support. FIG. 12 is a flowchart showing the processing flow of the meeting support system according to the present embodiment. FIG. 13 is a diagram showing an example of setting external feature points in the second embodiment. FIG. 14 is a diagram showing an example of the relationship between a shooting point and a meeting place in the fourth embodiment. FIG. 15 is a diagram explaining the movement of an information terminal in the sixth embodiment. FIG. 16 is a diagram explaining the image matching process in the sixth embodiment. FIG. 17 is a diagram explaining the relationship between a terminal local coordinate system and a terminal world coordinate system. FIG. 18 is a diagram explaining an example of guidance display in the sixth embodiment. FIG. 19 is a diagram explaining another example of guidance display in the sixth embodiment. FIG. 19 is a flowchart showing the processing flow in the sixth embodiment. FIG. 19 is a diagram explaining processing of Modification 1 of the sixth embodiment. FIG. 19 is a diagram explaining processing of Modification 2 of the sixth embodiment. FIG. 19 is a diagram explaining a use case of the guidance system according to the seventh embodiment. FIG. 19 is a diagram showing an example of a landmark image. FIG. 19 is a diagram showing an example of information accompanying the landmark image. FIG. 19 is a flowchart showing the processing flow of the guidance system in the seventh embodiment. FIG. 19 is a diagram explaining Modification 1 of the seventh embodiment. FIG. 19 is a diagram explaining Modification 2 of the seventh embodiment.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same components are designated by the same reference numerals throughout the drawings, and redundant explanations will be omitted.

[0010] In this embodiment, the in-vehicle terminal mounted on a vehicle and the portable information terminal carried by a user are collectively referred to as the information terminal. When a user uses a portable information terminal while on a vehicle, it corresponds to the in-vehicle terminal.

[0011] The information terminal according to this embodiment is characterized in that a processor installed in a terminal (hereinafter referred to as an "information terminal") used by a user heading to a meeting place calculates a terminal-to-terminal positional relationship between the information terminal and the other person's information terminal present at a destination point, using information relating to an external world image and external world feature points in the external world image acquired from an information terminal operated by another person (hereinafter referred to as an "other person's information terminal"), and information relating to an external world image captured by a camera installed in the mobile information terminal and external world feature points in the external world image, and outputs guidance information indicating the destination point to the mobile information terminal based on the terminal-to-terminal positional relationship. Therefore, the present invention can increase the commercial value of information processing devices to which the present invention is applied, and is therefore expected to contribute to Goal 8.2 of the Sustainable Development Goals (SDGs) proposed by the United Nations ("SDGs") ("Increase economic productivity through diversification, technological improvement, and innovation, particularly in industries that increase the value of goods and services and in labor-intensive industries").

[0012] The following describes an embodiment of the present invention, exemplifying an embodiment in which the present invention is applied to a meeting support system between a user of an information terminal and another person with whom the user has a meeting. Note that the information terminal may be simply referred to as a terminal. While approaching the meeting place, the information terminal user uses the information terminal to capture an external image of the area near the meeting place from a short distance away. The information terminal then generates external world feature point information indicating the relative positional relationship between the external world feature points in the external world image and the information terminal. Meanwhile, the other person uses the other person's information terminal, generally at the meeting place, to capture an external world image of the area near the meeting place. The other person's information terminal then generates external world feature point information indicating the relative positional relationship between the external world feature points in the external world image and the other person's information terminal. The external world images and external world feature point information of the two terminals are compared to determine the inter-terminal positional relationship, which is the relative positional relationship between the two terminals. The information terminal then guides the user to the meeting place based on this inter-terminal positional relationship information. In this case, the meeting place corresponds to the destination point to which the user of the information terminal is heading. The destination point is stationary in the world coordinate system. The user moves toward the destination point together with the information terminal.

[0013] First Embodiment FIG. 1 is a configuration diagram of a meeting support system 100 according to this embodiment.

[0014] An example of an information terminal is an in-vehicle terminal 1A. While approaching the meeting place, the in-vehicle terminal 1A captures an external image of the vicinity of the meeting place and generates external feature point information indicating the relative positional relationship between the external feature points in the external image and the information terminal. The other person waiting at the meeting place carries an HMD 1B or a smartphone 1C, which is an information terminal of the other person.

[0015] The in-vehicle terminal 1A, the HMD 1B, and the smartphone 1C are connected to one another via a network 3. Furthermore, the in-vehicle terminal 1A, the HMD 1B, and the smartphone 1C are connected to a server 2 via the network 3. The server 2 is a server that provides a meeting support service, and may be a cloud server or an edge server.

[0016] The other person uses the other person's information terminal to capture an external world image near the meeting place, creates external world feature point information indicating the relative positional relationship between the external world feature points in the external world image and the other person's information terminal, and transmits this information to the vehicle-mounted terminal 1A. The vehicle-mounted terminal 1A determines the other person's position at the meeting place using the received external world image and external world feature point information, the external world image captured by the vehicle-mounted terminal 1A, and the created external world feature point information.

[0017] The in-vehicle terminal 1A includes a processor connected to in-vehicle equipment such as an in-vehicle camera, a car navigation system, and a GPS mounted on the vehicle 1. For example, the in-vehicle terminal 1A may be configured by connecting a smartphone operated by a passenger in the vehicle 1, such as the driver, to in-vehicle equipment such as an in-vehicle camera using Bluetooth (registered trademark). The in-vehicle terminal 1A may also be configured by connecting the in-vehicle equipment and the processor via a vehicle network (CAN). Furthermore, the in-vehicle terminal 1A may be configured by a standalone mobile information terminal such as a smartphone operated by a passenger in the vehicle 1.

[0018] The information terminal may be an HMD 1B or a smartphone 1C. That is, the information terminal refers to a terminal carried by a user who is moving toward a target location, which is a meeting place, and the type of device used may be the in-vehicle terminal 1A, the HMD 1B, or the smartphone 1C.

[0019] Similarly, the other person's information terminal means a terminal for specifying a target point, and is not limited to the HMD 1B or the smartphone 1C, but may also be the in-vehicle terminal 1A.

[0020] FIG. 2 is a diagram showing the hardware configuration of the information terminal.

[0021] 2 shows the components necessary to realize the functions of the information terminal according to this embodiment and the components that are often typically installed. Furthermore, since the configurations of the information terminals used in this embodiment are similar to each other, they are described together. The configurations of the in-vehicle terminal 1A, HMD 1B, and smartphone 1C are distinguished by reference numerals A, B, and C, respectively. The in-vehicle terminal 1A, HMD 1B, and smartphone 1C may be collectively referred to as the terminal.

[0022] The in-vehicle terminal 1A, the HMD 1B, and the smartphone 1C include outer cameras 111A, 111B, and 111C, distance measurement sensors 113A, 113B, and 113C, position and orientation sensors 135A, 135B, and 135C, geomagnetic sensors 117A, 117B, and 117C, positioning sensors 118A, 118B, and 118C, and a real-time clock (RTC) The components include clocks 114A, 114B, and 114C, displays 119A, 119B, and 119C, microphones 121A, 121B, and 121C, speakers 122A, 122B, and 122C, processors 125A, 125B, and 125C, memories 128A, 128B, and 128C, and network communicators 120A, 120B, and 120C, which are connected to one another via buses 140A, 140B, and 140C that connect the components together.

[0023] Specific examples of the distance measurement sensors 113A, 113B, and 113C include a TOF (Time of Flight) sensor and a LiDAR (Laser Imaging Detection and Ranging) sensor.

[0024] The network communicators 120A, 120B, 120C are connected to antennas 123A, 123B, 123C that transmit and receive network communication signals.

[0025] The displays 119A, 119B, and 119C and the speakers 122A, 122B, and 122C correspond to output devices, and the microphones 121A, 121B, and 121C correspond to input devices.

[0026] The position and orientation sensors 135A, 135B, and 135C are used to measure the position and orientation of the terminal. Here, orientation refers to the orientation of the terminal in the real space in which the terminal exists. Specific configuration examples of the position and orientation sensors 135A, 135B, and 135C may include position posture cameras 136A, 136B, and 136C (hereinafter abbreviated as PP cameras), position posture distance measurement sensors 137A, 137B, and 137C (hereinafter abbreviated as PP distance measurement sensors), acceleration sensors 115A, 115B, and 115C, and gyro sensors 116A, 116B, and 116C.

[0027] The PP cameras 136A, 136B, and 136C may be the same as the outer cameras 111A, 111B, and 111C.

[0028] Furthermore, the PP distance measuring sensors 137A, 137B, and 137C may be the same as the distance measuring sensors 113A, 113B, and 113C.

[0029] Feature points that exist in the external world (hereinafter referred to as "external world feature points") are used to measure the position and orientation using the PP cameras 136A, 136B, and 136C and the PP ranging sensors 137A, 137B, and 137C. A plurality of external world feature points are used, and for example, external world feature points inside a room or external world feature points on a building outdoors may be used. As a preparation, the coordinate values ​​of the external world feature points are determined from the direction and distance measurements of the external world feature points. If the coordinate values ​​of the external world feature points in the external world coordinate system are known, this preparation can be omitted.

[0030] Then, by measuring changes in the direction and distance of the external feature points as seen from the terminal, changes in the position and orientation of the terminal from the relative positional relationship with the initial external feature points are measured. When measuring distance using a stereo camera method, only PP cameras 136A, 136B, and 136C may be used.

[0031] Note that while grasping the vertical direction is important in practical applications, if the coordinate values ​​of the external feature points in the external coordinate system (world coordinate system) and the vertical direction in the external coordinate system are known, the relative positional relationship between the terminal housing attitude and the vertical direction can be grasped by measuring the external feature points using PP cameras 136A, 136B, and 136C and PP ranging sensors 137A, 137B, and 137C. That is, for example, by measuring the coordinate values ​​of three external feature points in the terminal local coordinate system, the relationship between the numerical representations of two directional vectors between the external feature points in the external coordinate system and the terminal local coordinate system can be determined, and a rotation transformation that aligns the orientation of the terminal local coordinate system with the external coordinate system can be obtained. This determines the relationship between the terminal housing and the vertical direction.

[0032] Measurement of position and attitude using acceleration sensors 115A, 115B, and 115C and gyro sensors 116A, 116B, and 116C is performed by integrating acceleration and angular acceleration. This allows for determination of displacement from a reference position and reference attitude. Furthermore, measurement of the gravitational acceleration vector by acceleration sensors 115A, 115B, and 115C also allows for determination of the terminal's housing attitude and its relative position in the vertical direction.

[0033] The position and attitude of the terminal may be measured by referring to both the measurements using the PP cameras 136A, 136B, 136C and the PP ranging sensors 137A, 137B, 137C and the measurements using the acceleration sensors 115A, 115B, 115C and the gyro sensors 116A, 116B, 116C, or by using measurements from other types of sensors.

[0034] Positioning sensors 118A, 118B, and 118C are used to measure the indoor and outdoor positions of terminals. Specific examples of indoor positioning sensors include those that use beacon signals and location indicator marks. Outdoor positioning sensors include, for example, GPS (Global Positioning System) receivers that use signals from satellites.

[0035] The network communicators 120A, 120B, and 120C are communication interfaces that perform wireless communication between at least the terminal and the access point 4 via short-range wireless communication, wireless LAN, or base station communication, and include communication processing circuits corresponding to various predetermined communication interfaces and are connected to the antennas 123A, 123B, and 123C. The network communicators 120A, 120B, and 120C transmit and receive biometric information, video data, control signals, and the like. Note that short-range wireless communication is performed using a wireless LAN such as Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or Wi-Fi (registered trademark). Furthermore, for base station communication, long-distance wireless communication such as LTE (Long Term Evolution, registered trademark), LTE-Advanced, Mobile WiMAX (Worldwide Interoperability for Microwave Access, registered trademark), or WiMAX2 may be used.

[0036] Displays 119A, 119B, and 119C are installed in a manner that allows the terminal users to view images. Processors 125A, 125B, and 125C control the operation of the terminals.

[0037] The processors 125A, 125B, and 125C are configured by, for example, a CPU.

[0038] The memories 128A, 128B, and 128C are configured by flash memory and nonvolatile memory. The memories 128A, 128B, and 128C also store programs 126A, 126B, and 126C such as an OS (Operating System) and operation control applications, and data 127A, 127B, and 127C used by the processors 125A, 125B, and 125C.

[0039] The processors 125A, 125B, 125C load the programs 126A, 126B, 126C into the memories 128A, 128B, 128C and execute them, and read out the data 127A, 127B, 127C as needed and use it in the execution processing of the programs 126A, 126B, 126C.

[0040] The smartphone 1C further includes, as components, an inner camera 112C, a touch panel 124 as an input interface, and a telephone network communication device 131C.

[0041] FIG. 3 is a hardware configuration diagram of the server.

[0042] In the server 2, a processor 225, a memory 228, and a network communicator 220 are connected to one another via a bus 240 that connects the various components. The network communicator 220 is connected to an antenna 223 that transmits and receives network communication signals.

[0043] The processor 225 controls the operation of the server and is configured by, for example, a CPU.

[0044] The memory 228 is configured by a flash memory and a nonvolatile memory, and also stores programs 226 such as an OS (Operating System) and an operation control application, and data 227 used by the processor 225.

[0045] The processor 225 loads the program 226 into the memory 228 and executes it, and reads out the data 227 as necessary to use it in the execution process of the program 226 .

[0046] [Processing Flow] FIG. 4 is a diagram showing an example of the usage status of the meeting support system 100 according to this embodiment.

[0047] Assume that a passenger and a taxi are meeting at the location where the passenger is standing in the figure. The passenger corresponds to a third party who specifies the destination point, which is the meeting place. The taxi corresponds to vehicle 1 in Figure 1, and the taxi driver corresponds to a user traveling toward the destination point, which is the meeting place.

[0048] The taxi is equipped with an in-vehicle terminal 1A, and the passenger carries an HMD 1B. The in-vehicle terminal 1A corresponds to the user information terminal. The HMD 1B corresponds to the other person's information terminal. The following process is basically the same even if the other person's information terminal is a smartphone 1C. To know the relative positional relationship between the terminals, a video of the vicinity of the meeting place is taken. The video may be a video or a still image. The relative positional relationship between the terminals may be referred to as the inter-terminal positional relationship.

[0049] Fig. 5 is a diagram showing an image captured from a user information terminal, and Fig. 6 is a diagram showing an image captured from another person's information terminal. The first external world image 1000 shown in Fig. 5 includes first external world feature points 1010A, 1010B, and 1010C. Similarly, the second external world image 1001 shown in Fig. 6 includes second external world feature points 1011A, 1011B, and 1011C. Because it is necessary for both terminals to capture roughly the same location, approximate data on the subject position, shooting location, and shooting direction may be communicated from one terminal to the other terminal or bidirectionally using positioning data from a general-purpose positioning system such as GPS that can determine the approximate position.

[0050] (Method for acquiring the positional relationship between terminals (1)) First, we will explain the terms related to coordinate systems. First, a "world coordinate system" is a coordinate system that is stationary relative to the external world, and a "local coordinate system" is a coordinate system that is not stationary relative to the external world but is fixed to some kind of object. Then, a "terminal world coordinate system" is a coordinate system that is fixed relative to the external world and is used by the terminal under consideration, and a "terminal local coordinate system" is a coordinate system that is fixed to the housing of the terminal under consideration.

[0051] Each terminal requires some kind of "world coordinate system" to describe its own position and orientation relative to the external world as its control data. The "world coordinate system" used by each terminal is called the "terminal world coordinate system." Generally, the "terminal world coordinate system" differs from terminal to terminal. The core of the present invention is a technology described in "Methods (1) and (2) for Acquiring Positional Relationship Between Terminals," which compares external world images captured by two terminals and determines coordinate transformation parameters between the two terminal world coordinate systems. These coordinate transformation parameters determine the position coordinates of the other terminal within the terminal world coordinate system, or vice versa. The difference between acquisition methods (1) and (2) is the accompanying information of the external world image. In using external world images, the present invention falls under the category of VPS (Visual Positioning System) technology. However, the present invention differs from conventional VPS technology in that the amount of image information required to meet the objectives of each embodiment is small.

[0052] The "terminal local coordinate system" is used to describe the positions of external feature points and the direction as seen from the terminal, based on the terminal's position and orientation.

[0053] In the explanation, we sometimes say "numerical expression in a coordinate system." We also sometimes shorten it to "expression in a coordinate system," but in this case "expression" means "numerical expression." Specifically, when we say "numerical expression" of a point or position in the outside world, we are referring to the coordinate value of the point or position in a certain coordinate system. If we express it in terms of components for each coordinate axis in a three-dimensional coordinate system, it is as follows: (P Ｘ , P Ｙ , P Ｚ ) is data such as this. Direction, or the "numerical expression" of a directional vector, which is the same thing, is a numerical representation of a directional vector in a certain coordinate system. If expressed in components for each coordinate axis in a three-dimensional coordinate system, it is expressed as (V Ｘ , V Ｙ , V Ｚ) The specific values ​​of these "numerical representations" change when the coordinate system changes, even if they represent the same point or the same directional vector. In the process described below, the specific values ​​of the "numerical representation" of the same point or the same directional vector change depending on the respective "terminal world coordinate systems," so conversion parameters are found to align the "numerical representation" between different "terminal world coordinate systems" based on these differences. Knowing the position of the other terminal means knowing the "numerical representation" of the other terminal's position in the "terminal world coordinate system" of the own terminal.

[0054] Next, we will explain the symbols. In the following explanation, unless otherwise specified, when rotation, direction vectors, and vectors (including point position indications) are represented by a single symbol, they will be represented as quaternions. In addition, the norm of both rotation and direction vectors is normalized to 1. Furthermore, when a direction vector or vector is represented by a quaternion, the real part of the quaternion is set to 0. We will first organize the symbols used in the explanation.

[0055] First, the basis elements of a quaternion other than 1 are represented as i, j, and k, and unless otherwise specified, when representing a direction vector or vector, the relationship to the axis directions of the coordinate system under consideration is X axis: i, Y axis: j, and Z axis: k. For example, when the basis element i in quaternion notation represents a vector, this vector means (1, 0, 0) in the component notation for each of the coordinate axes X, Y, and Z. In each three-dimensional coordinate system, the coordinate axes are represented by a combination of the letters X, Y, and Z and their subscripts.

[0056] A quaternion Q that rotates a direction vector n by an angle λ in the direction of a right-handed screw. Ｒ (n, λ) is expressed by the following formula: Q Ｒ (n, λ) = cos(λ / 2) + sin(λ / 2)n (1) Note that rotating in the direction in which a right-handed screw advances means rotating in the direction in which the screw rotates when a right-handed screw advances (clockwise when looking in the direction in which the screw advances).

[0057] Here, Q ＊ is the conjugate quaternion of Q, the order of multiplication when vector a is rotated to vector b by rotation Q is as follows: b = QaQ ＊ ... (2)

[0058] Specifically, the quaternion that rotates 90 degrees around the direction vector n in the direction of n, in the direction of a right-handed screw, is expressed as H(n). H(n) is expressed as follows: H(n) = 2 －１／２ +2 －１／２ n... (3)

[0059] [·] is an operator that normalizes a non-zero quaternion to norm 1. If q is any non-zero quaternion, [·] is defined as follows: [q] = q(qq ＊ ) －１／２ ...(4)

[0060] P Ｔ (n) is an operator that projects a vector onto a plane perpendicular to the direction vector n. n is assumed to be normalized. The projection of vector p is expressed by the following equation: P Ｔ (n) p = p + nS Ｃ (np) (5) Here, Sc(q) denotes the real part of the quaternion q.

[0061] Together with the above symbols, the direction vector n １ and direction vector n ２ Rotate around an axis perpendicular to the plane containing n １ to n ２ Rotation Q to be superimposed １２ Use equation (6) to find Q １２ = [1-n ２ n １ ]・・・(6)

[0062] It can be easily confirmed that the formula (6) holds true by directly calculating the right-hand side, taking care to set the real part of the direction vector to 0.

[0063] Now, in order to match the external feature points in the external image of the area around the meeting point taken by each device, external feature point information for each image is created. The image can be a still image or a video. In the case of a video, at least one frame of the image is extracted and used.

[0064] The external world feature point information is list information that includes the positions of the external world feature points in the video and coordinate value data in the terminal's world coordinate system, corresponding to the captured video. An example of the external world feature point information is shown in FIG. 7A. The coordinate values ​​in FIG. 7A are written in normal component notation. Such external world feature point information is created for each of the videos from both terminals. The coordinate values ​​of the matched external world feature points may be obtained later using the feature point information or external world feature point matching, which will be described later.

[0065] When using the method (2) for acquiring the positional relationship between terminals described later, one of the terminals uses the value of a direction vector in the terminal world coordinate system pointing from the terminal to the external world feature point in place of the coordinate value of the external world feature point in the external world feature point information.

[0066] Here, the position on the video refers to coordinates on the video, for example, coordinates with the upper left corner of the video as the origin, the horizontal direction with the right direction as the positive direction as the U coordinate, and the vertical direction with the downward direction as the positive direction as the V coordinate ( FIG. 8 ). In the following, the coordinate system refers to a world coordinate system fixed to the external world unless otherwise specified. The external world feature information also includes the feature amounts of the external world feature points, which will be described later. Using these feature amounts as clues, external world feature point matching, which will be described later, is performed to determine the correspondence between the external world feature points in the video of both devices. An example of an external world feature point matching table that summarizes this correspondence is shown in FIG. 7B .

[0067] First, as a method (1) for acquiring the positional relationship between terminals, a case will be described in which the external world feature point information of both terminals contains coordinate value data of the external world feature point. To obtain the coordinate value of the external world feature point in the terminal world coordinate system, first, the outer cameras 111A and 111B measure the direction vector of the external world feature point as seen from the terminal position relative to the terminal housing, and the ranging sensors 113A and 113B measure the distance from the terminal position to the external world feature point to obtain the coordinate value in the terminal local coordinate system. Next, the position and orientation sensors 135A and 135B measure the position and orientation of the terminal in the world coordinate system, and the coordinate value in the terminal local coordinate system is converted to a coordinate value in the terminal world coordinate system, thereby obtaining the distance. Distance measurement may also be performed using a moving stereo method using the outer cameras 111A and 111B.

[0068] Here, the method for calculating the coordinate values ​​of the external feature points will be described in detail. First, the terminal local coordinate system {X Ｌ Axis, Y Ｌ axis, Z Ｌ Axis, origin: O Ｌ} is defined (Fig. 8). Then, the unit vectors in the directions of the coordinate axes are defined as N ＸＬ , N ＹＬ , N ＺＬ The specific notation in quaternions is N ＸＬ = i, N ＹＬ = j, N ＺＬ = k, where i, j, and k are the basis elements of the quaternion. The origin of the terminal local coordinate system is the point where the direction of the incident light from the outside world converges. The optical axis direction is X Ｌ axis, and the horizontal direction of the image is perpendicular to the optical axis and points leftward in the Y direction. Ｌ Axis, vertical direction of the image is Z Ｌ The axis is X. Ｌ The axis passes through the center of the image. Ｗ Axis, Y Ｗ axis, Z Ｗ Axis, origin: O Ｗ} is defined to recognize the position and orientation of the terminal. As an example of the definition of the terminal world coordinate system, there is a method of fixing the terminal local coordinate system at the time of terminal initialization to the external world as it is and using it as the terminal world coordinate system. The maximum value of the video coordinate value is (U Ｍ , V Ｍ ) and the horizontal angle of view is μ Ｕ , vertical angle of view is μ Ｖ , the image plane is perpendicular to the optical axis, and distortion caused by the optical system is also corrected.

[0069] Now, the position of the external feature point on the image (U Ｐ , V Ｐ ) The origin of the terminal local coordinate system O Ｌ The distance from the center of the image to the center of the image is f, and the coordinate value on the image (U Ｐ , V Ｐ ) is the position of the point on the image represented by P Ｎ Then, P Ｎ The three-dimensional expression of P in the terminal local coordinate system is expressed by the following equation: Ｎ = fN ＸＬ +fS Ｎ ... (7)

[0070] Here, S Ｎ is expressed as follows in three-dimensional terms in the terminal local coordinate system: Ｎ = ((U Ｍ -2U Ｐ ) / 2U Ｍ ) tan(μ Ｕ / 2) N ＹＬ +((V Ｍ -2V Ｐ ) / 2V Ｍ ) tan(μ Ｖ / 2) N ＺＬ ...(8)

[0071] From this, the origin of the terminal local coordinate system O Ｌ The external feature point direction vector N Ｒ is given by the following equation in numerical expression in the terminal local coordinate system: Ｒ = [N ＸＬ +S Ｎ ]・・・(9)

[0072] The external feature point direction vector N Ｒ and the distance d from the terminal position to the external feature point, which is calculated separately. Ｒ multiplied by , the numerical representation R of the coordinates of the external feature points in the terminal local coordinate system is obtained. Ｌ is obtained. R Ｌ = d Ｒ N Ｒ ...(10)

[0073] Representation of external feature point coordinate values ​​in the device local coordinate system R Ｌ From the expression R in the terminal world coordinate system Ｗ To convert to this, information on the orientation and origin position of the terminal local coordinate system in the terminal world coordinate system is used. The orientation and origin position of the terminal local coordinate system are known because the changes from the initialized state are measured by the position and orientation sensors 135A and 135B. The rotation of the terminal local coordinate system in the terminal world coordinate system from the state where the directions of the X axes, Y axes, and Z axes are aligned to the current point in time is expressed as Q. ＬＷ The coordinate value of the origin of the terminal local coordinate system in the terminal world coordinate system is O ＬＷ Then, R Ｗ is calculated as follows: R Ｗ = QＬＷ R Ｌ Q ＬＷ ＊ +O ＬＷ ...(11)

[0074] In the method (2) for acquiring the positional relationship between terminals, which will be described later, the external feature point information of one terminal uses a direction vector from the terminal position to the external feature point instead of the coordinate value of the external feature point. The numerical expression in the terminal world coordinate system of the direction vector from the terminal position to the external feature point is Ｗ Then, N Ｗ is calculated as follows: N Ｗ = Q ＬＷ N Ｌ Q ＬＷ ＊ ...(11')

[0075] Examples of external field feature points are shown in Figures 5 and 6. Here, the corners of window frames of three buildings are extracted as external field feature points as examples of stationary objects. To determine the positional relationship between terminals, three external field feature points that are not on the same line are required. Since there is no guarantee that individual external field feature points can be matched between external field images captured by both terminals with 100% probability, it is desirable to list three or more external field feature points in the external field feature point information. In Figures 5 and 6, the matched external field feature points are shown as first external field feature points 1010A, 1010B, and 1010C and second external field feature points 1011A, 1011B, and 1011C.

[0076] 5 is an example of an external world image captured by a user information terminal, which is designated as a first external world image 1000, and FIG. 6 is an example of an external world image captured by another person's information terminal, which is distinguished from a second external world image 1001. First external world feature points 1010A, 1010B, and 1010C are captured in the first external world image 1000, and second external world feature points 1011A, 1011B, and 1011C are captured in the second external world image 1001. Here, for example, an external world feature point captured in the first external world image is referred to as a first external world feature point, but if there is no need to specify in which image the referenced external world feature point was captured, the expressions specifying the image, such as first or second, may be omitted.

[0077] The distinction between first and second coordinate systems for the external world image may also be used for the terminal coordinate system. That is, the world coordinate system used by the user information terminal may be referred to as the first terminal world coordinate system, and the world coordinate system used by the other person's information terminal may be referred to as the second terminal world coordinate system.

[0078] Here, a method for matching external feature points in the external image of both terminals will be described. First, external feature points, mainly corners such as window frames of buildings, are extracted from the external image. Next, feature quantities of the extracted external feature points are calculated from the external image. There are various methods for calculating feature quantities. For example, SIFT (Scale-Invariant Feature Transform), SURF (Speed-Up Robust Features), BRIEF (Binary Robust Independent Elementary Features), etc., and any feature quantity may be used for matching.

[0079] Here, an overview of SIFT features is explained (Fig. 9). First, the maximum gradient direction of the brightness gradient near the external feature point is found, and the external image area is divided into 16 4x4 blocks near the feature point according to this direction. Then, the brightness gradient in each block is represented as a histogram in eight directions. The data of this histogram is collected and represented as a 128-dimensional vector value, which is the SIFT feature.

[0080] Next, an example of a procedure for matching feature points based on feature amounts will be described. While Figs. 5 and 6 show first and second external world feature points whose correspondence relationship is determined as a result of the matching process, it is not clear which first and second external world feature points represent the same external world feature point until the result of the matching process is obtained. Therefore, in the following description of the processing procedure, the labels of the first and second external world feature points are independent. The label of the first external world feature point during the matching process is referred to as a ｈ (h=1, 2, 3...M), and the label of the second external feature point is b ｈ (h=1, 2, 3...N). The first external feature point a r1 In order to find the second external feature point corresponding to the first external feature point a r1The distance between the feature vectors of the second external feature point b is calculated. Here, the distance between the feature vectors is the norm of the difference between the two feature vectors. Then, the second external feature point b, which has the smallest inter-feature distance, is calculated. r2 the first external feature point a r1 is the corresponding second external feature point, and matching is established. However, if this smallest inter-feature distance is 0.8 times or more the second smallest inter-feature distance, it is determined that there is no corresponding second external feature point, and matching is not established. This is because the existence of another second external feature point with a close inter-feature distance may indicate an incorrect correspondence.

[0081] After the first stage of matching, a cross-check is performed. First, in the first stage of matching, the first external feature point a r1 The second external feature point b r2 Next, the second external feature point b r2 is fixed, and an external feature point is selected from the first external feature point by the above matching method. If there is no first external feature point that matches in this reverse matching, the first external feature point a r1 and the second external feature point b r2 If there is a first external feature point for which matching is established in the reverse direction, the first external feature point is set as a r3 This first external feature point a r3 is the first external feature point a r１ If it is equal to, the first external feature point a r１ and the second external feature point b r2 If the first external feature point a is not equal to the first external feature point a, the first external feature point a is considered to be equal to the first external feature point a. r１ and the second external feature point b r２ is determined to be unsuccessful. Note that the first external world feature point and the second external world feature point for which matching has been achieved may be managed by assigning the same label. In the following description, the external world feature point α, the external world feature point β, and the external world feature point γ will be used as examples of the external world feature points for which matching has been achieved in a pair of feature points in the set of first external world feature points and feature points in the set of second external world feature points.

[0082] If the number of matching external feature points is less than a predetermined number, the external areas captured by the two terminals may be different, so the video may be re-captured.

[0083] The extraction of external world feature points, the calculation of feature amounts, and the execution of matching may be performed by either terminal. The extraction of external world feature points and the calculation of feature amounts may be performed by each terminal, and the calculation results of feature amounts may be collected on one terminal to execute matching. Alternatively, external world video information may be sent from one terminal to the other, and the subsequent processing up to matching may be executed on one terminal. Furthermore, the calculation processing may be performed by server 2.

[0084] Furthermore, the measurement of the coordinate values ​​of the external world feature points may be performed for all the extracted external world feature points in the external world image, or may be limited to some of the external world feature points required for subsequent processing. When the terminal is moving, in order to obtain the coordinate values ​​of the external world feature points after the timing of capturing the external world image used for external world feature point matching, it is sufficient to capture the external world image from different capturing positions, measure the distance to the external world feature point using a moving stereo method, and obtain the coordinate values.

[0085] From among the external feature points for which matching has been established between the external image of both terminals in the processing up to this point, three external feature points that are not on the same line are selected, and a process is performed to determine the relative positional relationship between the two terminals.

[0086] FIG. 10 is a diagram showing the contents of a method (1) for acquiring the inter-terminal positional relationship of another person's information terminal with respect to a user information terminal, and is an explanatory diagram of two direction vectors defined in real space.

[0087] As shown in FIG. 10, N １ , N ２These two direction vectors can be defined, and the rotation for matching the orientations of the two terminal world coordinate systems can be calculated from the numerical representation of these two direction vectors in the terminal world coordinate systems of the in-vehicle terminal 1A and the HMD 1B. Also, the difference in the origins of the two terminal world coordinate systems can be calculated from the numerical representation of any one point in real space in the terminal world coordinate systems. Information on this rotation for matching the orientations and the difference in origins is conversion information between the terminal world coordinate systems of the in-vehicle terminal 1A and the HMD 1B. If conversion between coordinate systems can be performed, it is naturally possible to know the position of the other terminal.

[0088] In addition, N in the real space １ , N ２ If the direction vector can be measured and recognized by another method, the number of external feature points to be used can be reduced, and this method will be described later as a modified example.

[0089] FIG. 11 is a diagram showing the contents of a method (1) for acquiring the inter-terminal positional relationship of an other information terminal with respect to a user information terminal, and is a diagram showing two direction vectors in the terminal world coordinate systems of the user information terminal and the other information terminal.

[0090] Direction vector N １ , N ２ The numerical expression in the terminal world coordinate system of the vehicle-mounted terminal 1A is N （Ａ） １ , N （Ａ） ２ and the numerical expression in the terminal world coordinate system of the HMD 1B is N （Ｂ） １ , N （Ｂ） ２ These direction vectors are shown in Fig. 11. Here, the space in Fig. 11 is the coordinate space of the vehicle-mounted terminal 1A, and N （Ｂ） １ , N （Ｂ） ２ When expressed in numerical terms in the coordinate system of the vehicle-mounted terminal 1A, N （Ａ） １ , N （Ａ） ２ The direction vector is different from that of （Ｂ） １ , N （Ｂ） ２ Rotate to N （Ａ）１ , N （Ａ） ２ This explains the transformation that is superimposed on

[0091] First, N （Ａ） １ and N （Ｂ） １ Rotate around an axis perpendicular to the plane containing N （Ｂ） １ N （Ａ） １ Such a rotation Q １ is expressed by the following equation (12): １ = [1-N （Ａ） １ N （Ｂ） １ ]・・・(12)

[0092] In addition, from the beginning, N （Ｂ） １ N （Ａ） １ In the special case where １ = 1, i.e. no rotation.

[0093] After this rotation, N （Ｂ） ２ The direction vector of N （Ｂ）＃ ２ If you write this, （Ｂ）＃ ２ is expressed by the following equation (13): （Ｂ）＃ ２ =Q １ N （Ｂ） ２ Q １ ＊ ...(13)

[0094] Next, N （Ｂ）＃ ２ N （Ａ） １ Rotate around the axis and （Ａ） ２ The rotation at this time is Q ２ When written as Q ２ is expressed as the following equation (14): Q ２ = [1-[P Ｔ (N （Ａ） １ ) N （Ａ） ２ ][P Ｔ (N（Ａ） １ ) N （Ｂ）＃ ２ ]]...(14) where P Ｔ (n) is an operator that projects a vector onto a plane perpendicular to the direction vector n (defined in equation (5)). （Ｂ）＃ ２ N （Ａ） ２ In the special case where ２ = 1, i.e. no rotation.

[0095] From the above, the numerical expression of the direction vector expressed in the terminal world coordinate system of HMD1B is rotation (Q ２ Q １ ) can be converted into a numerical expression of a direction vector expressed in the terminal world coordinate system of the vehicle-mounted terminal 1A. For the following explanation, rotation Q ＢＡ is defined by the following equation (15): ＢＡ =Q ２ Q １ ...(15)

[0096] Next, the difference between the origins of the two terminal world coordinate systems is calculated. ＢＡ is expressed by the following equation (16): ＢＡ =D ＃（Ａ） -Q ＢＡ D ＃（Ｂ） Q ＢＡ ＊ ...(16)

[0097] Here, D ＃（Ａ） , D ＃（Ｂ） are numerical representations of arbitrarily selected external points in the terminal world coordinate system of the vehicle-mounted terminal 1A and the terminal world coordinate system of the HMD 1B. For example, if an external feature point α is selected and D ＃（Ａ） , D ＃（Ｂ） If you substitute specific values ​​for , ＢＡ is obtained. In this way, O ＢＡ If we use the above formula, the conversion of numerical representation between coordinate systems of any point is given by the following formula (17): （Ａ） =Q ＢＡ D （Ｂ） Q ＢＡ ＊ +O ＢＡ...(17)

[0098] The inverse transformation is given by the following equation (18): D （Ｂ） = Q ＢＡ ＊ D （Ａ） Q ＢＡ -Q ＢＡ ＊ O ＢＡ Q ＢＡ ...(18)

[0099] Here, D （Ａ） , D （Ｂ） are numerical representations of an arbitrary point in the coordinate system of the vehicle-mounted terminal 1A and the coordinate system of the HMD 1B, respectively.

[0100] The rotation Q, which is the transformation parameter between the coordinate systems of both terminals, is calculated as follows: ＢＡ and the difference between the origin position O ＢＡ By sharing this information between both terminals, each terminal can obtain the position coordinates of the other terminal in its own coordinate system and convert them into coordinates in its own coordinate system. In other words, both terminals can grasp the positional relationship between the terminals. Alternatively, one terminal may perform the coordinate conversion calculation and share the calculation results.

[0101] Note that, by using the above method, when the coordinate values ​​of the external world feature points in the terminal world coordinate system are known, it is possible to determine the position and orientation of the terminal with respect to the terminal world coordinate system using PP cameras 136A, 136B, and 136C and PP ranging sensors 137A, 137B, and 137C. Since the external world feature points can be identified using PP cameras 136A, 136B, and 136C and the coordinate values ​​of the external world feature points in the terminal local coordinate system can be measured using PP ranging sensors 137A, 137B, and 137C, it is possible to determine the transformation parameters between the terminal world coordinate system and the terminal local coordinate system using the above method.

[0102] (Variation of Method (1) for Acquiring Positional Relationship Between Terminals) If both terminals can recognize a specific direction that allows them to recognize a direction, rather than a direction between points in real space, the number of required external feature points can be reduced.

[0103] For example, the vertically upward direction can be recognized by measuring the direction of gravitational acceleration using acceleration sensors 115A and 115B on both terminals. If this vertically upward direction is considered as one of the directions in the real space described above, the number of external feature points used can be two. In other words, the direction vector from external feature point α to β is N １ and the vertical upward direction vector is N ２ If so, the transformation formula for the coordinate systems between the two terminals can be found using the same calculation procedure.

[0104] Furthermore, as another specific direction in the real space, for example, the north direction is measured by the geomagnetic sensors 117A and 117B, and a direction vector N １ In order to find the difference between the origins of both coordinate systems using equation (16), the coordinate values ​​of one point in the external world must be known in both coordinate systems, so one external feature point is used.

[0105] The specific direction in real space is not limited to the above, and other directions may be used. For example, the direction of a landmark that is sufficiently far from the terminal may be used as the direction in real space.

[0106] As described above, by using real space directions other than the directions between external field feature points, it is possible to reduce the number of external field feature points used and the calculation load for external field feature point matching.

[0107] (Method (2) of acquiring positional relationship between terminals) Next, a processing method will be described when the coordinate values ​​of external world feature points are measured only on one terminal. The other terminal measures only the direction of the external world feature points. Here, it is assumed that the coordinate values ​​of the external world feature points are measured on the vehicle-mounted terminal 1A, but it is the same even if the coordinate values ​​of the external world feature points are measured on the HMD 1B. Note that when distance measurement is performed using a moving stereo technique or when using a processing procedure that does not measure distance as in this method (2) of acquiring positional relationship between terminals, a hardware configuration that does not use distance measurement sensors 113A, 113B and PP distance measurement sensors 137A, 137B is also possible.

[0108] The vehicle-mounted terminal 1A generates external world feature point information based on coordinate value data of the external world feature points, and the HMD 1B generates external world feature point information based on directional vector data to the external world feature points. The external world feature point matching is the same as the method used in the method (1) for acquiring the positional relationship between terminals.

[0109] 12 is a diagram showing the concept of separating the positional relationship between the matched external feature points and the HMD 1B, which is the other person's information terminal, into a horizontal projection plane and a vertical projection plane. A method for determining the vertical projection plane will be described later.

[0110] First, as shown in Fig. 12, the positional relationship between the external feature points and the HMD 1B, which is the other person's information terminal, is orthogonally projected onto a horizontal projection plane and a vertical projection plane, and the positional relationship within the horizontal plane and the positional relationship within the vertical plane are considered separately. The vertical direction required for projection onto the horizontal projection plane can be recognized by both the in-vehicle terminal 1A and the HMD 1B, and is performed by measuring the direction of gravitational acceleration using acceleration sensors 115A and 115B, for example.

[0111] 12, the terminal position X of the HMD 1B and the positions of the external world feature points α, β, and γ in the real space projected onto a horizontal projection plane are represented by terminal position X' and external world feature points α', β', and γ'. Also, the terminal position X of the HMD 1B and the positions of the external world feature points α, β, and γ projected onto a vertical projection plane are represented by terminal position X'' and external world feature points α'', β'', and γ''.

[0112] When a taxi approaches a passenger, the difference in vertical position between the taxi and the passenger is usually not large enough to cause inconvenience during the meeting. Therefore, if the relative positional relationship between the two in the horizontal projection plane is known, support for the meeting place can be provided, but the processing for a general case will also be explained. First, the processing content of the in-vehicle terminal 1A estimating the terminal position X' of the other person's information terminal in the horizontal projection plane will be explained.

[0113] FIG. 13 is a diagram showing the contents of a method (2) for acquiring the inter-terminal positional relationship between a user information terminal and another person's information terminal.

[0114] The coordinate values ​​of the external feature points α, β, and γ in the terminal world coordinate system of the vehicle-mounted terminal 1A are measured by the vehicle-mounted terminal 1A.

[0115] On the other hand, in the HMD 1B, the direction vectors from the terminal to the external feature points α, β, and γ are expressed numerically in the coordinate system of the HMD 1B, N α , N β , N γ Ask for.

[0116] Direction vector N to the external feature point α , N β , N γ Once this is determined, the vertical upward direction vector is N Ｖ and the angle of view (θ') between the external feature points as seen from the position of the HMD 1B (terminal position X') in the horizontal projection plane. αβ , θ' βγ , θ' γα ) is calculated from the dot product between each direction vector. Note that -SC(ab) represents the dot product of vector a and vector b, and each angle of view can be calculated using the following formula: θ' αβ = cos －１ (-Sc([P Ｔ (N Ｖ ) N α ][P Ｔ (N Ｖ ) N β ]))...(19) θ' βγ = cos －１ (-Sc([P Ｔ (N Ｖ ) N β ][P Ｔ (N Ｖ ) N γ ]))...(20) θ' γα = cos －１ (-Sc([P Ｔ (N Ｖ ) N γ ][P Ｔ (N Ｖ ) N α ])) ... (21)

[0117] According to the theorem of circular angles, the angle between the external feature point α' and the external feature point β' is θ' αβThe set of points where X' is the terminal position is a point somewhere on the circumference of this circle. The same can be said for other sets of feature points. Therefore, three circles are created in total, and the location of terminal position X' in the horizontal projection plane of HMD 1B can be determined as the intersection of two of these circles (Figure 13).

[0118] To determine the specific coordinate value of X', select a horizontal projection plane, which passes through β and is Ｖ A plane perpendicular to β is selected. In other words, β' is equal to β. A two-dimensional coordinate system is defined on this plane. Hereafter, it is called coordinate system A1. The origin is β. A direction vector N from β to α' is Ｔ１１ is one of the coordinate axes. Direction vector N Ｔ１１ is given by the following formula: N Ｔ１１ = [P Ｔ (N Ｖ )(α-β)]...(22)

[0119] And the direction vector N Ｔ１１ , N Ｖ A direction vector N rotated 90 degrees around the axis Ｔ１２ Let N be the other coordinate axis. Ｔ１２ is given by the following formula: N Ｔ１２ = H(N Ｖ ) N Ｔ１１ H (N Ｖ ) ＊ ...(23)

[0120] Here, H(n) is a 90 degree rotation around the axis of the direction vector n (defined in equation (3)).

[0121] The numerical expression in the above coordinate system A1 is represented by a superscript (A1). The numerical expression of an arbitrary point D in the coordinate system A1 can be expressed in component notation as follows: D （Ａ１） = (-Sc(N Ｔ１１ P Ｔ (N Ｖ )(D-β)), -Sc(N Ｔ１２ P Ｔ (N Ｖ )(D-β)))...(24)

[0122] α' （Ａ１） , β' （Ａ１） , γ'（Ａ１） The coordinate value and the angle of view (θ' αβ , θ' βγ , θ' γα ) has been found, the coordinate value X' can be calculated by the method described above for finding X' from the intersection of the circles. （Ａ１） The coordinate value X' thus obtained is （Ａ１） The component expression of X' is as follows: （Ａ１） = ((X' （Ａ１） ) １ , (X' （Ａ１） ) ２ ) ... (25)

[0123] Similarly, if the terminal position X'' of the HMD 1B in the vertical projection plane is found, the position of the vehicle-mounted terminal 1A of the HMD 1B in the terminal world coordinate system can be determined. Here, the horizontal direction vector N Ｈ It is necessary to determine the horizontal vector N Ｈ In this case, the vertically upward direction vector N Ｖ is a direction vector rotated 90 degrees around the axis of the direction vector from X' to β, and N Ｈ Define N Ｈ The numerical expression in each coordinate system is as follows: Ｈ （Ａ） = H([β （Ａ） -X' （Ａ） ]) Nv （Ａ） H ([β （Ａ） -X' （Ａ） ]) ＊ ... (26) N Ｈ （Ｂ） = H([P Ｔ (N Ｖ （Ｂ） ) N β （Ｂ） ]) Nv （Ｂ） H ([P Ｔ (N Ｖ （Ｂ） ) N β （Ｂ） ]) ＊ ...(27)

[0124] Here, [β' in formula (26) （Ａ） -X'（Ａ） ] is represented by the following formula, and X' （Ａ１） can be determined if [β （Ａ） -X' （Ａ） ]=-[(X' （Ａ１） ) １ N Ｔ１１ （Ａ） +(X' （Ａ１） ) ２ N Ｔ１２ （Ａ） ]・・・(28)

[0125] This N Ｈ By projecting each position onto the vertical projection plane using the above equation and performing the same processing as that performed within the horizontal projection plane, the position of the terminal position X'' within the vertical projection plane of the HMD 1B can be determined.

[0126] First, similarly to the case in the horizontal projection plane (FIG. 13), the angle of view (θ'') between the external feature points as seen from the position of the HMD 1B (terminal position X'') in the vertical projection plane is calculated. αβ , θ′′ βγ , θ′′ γα ) can be calculated using the following formula: θ'' αβ = cos －１ (-Sc([P Ｔ (N Ｈ ) N α ][P Ｔ (N Ｈ ) N β ]))...(29) θ'' βγ = cos －１ (-Sc([P Ｔ (N Ｈ ) N β ][P Ｔ (N Ｈ ) N γ ]))...(30) θ'' γα = cos －１ (-Sc([P Ｔ (N Ｈ ) N γ ][P Ｔ (N Ｈ ) N α ])) ... (31)

[0127] Then, similarly to the processing within the horizontal projection plane, the position of the terminal position X'' within the vertical projection plane of the HMD 1B can be determined as the intersection of the circles.

[0128] To find the specific coordinate value of X'', select one vertical projection plane. Here, the plane passing through β is N Ｈ Select a plane that is perpendicular to . In other words, β'' is equal to β. The vertical projection plane selected here is a plane obtained by rotating the vertical projection plane selected above by 90 degrees around the axis of the direction vector from X' to β. In other words, X' is located within this vertical projection plane.

[0129] Now, let us define a two-dimensional coordinate system on this plane. Hereafter, we will call it coordinate system A2. The origin is β. The direction vector N from β to α'' is Ｔ２１ is one of the coordinate axes. Direction vector N Ｔ２１ is given by the following formula: N Ｔ２１ = [P Ｔ (N Ｈ )(α-β)]...(32)

[0130] And the direction vector N Ｔ２１ , N Ｈ A direction vector N rotated 90 degrees around the axis Ｔ２２ Let N be the other coordinate axis. Ｔ２２ is given by the following formula: N Ｔ２２ = H(N Ｈ ) N Ｔ２１ H (N Ｈ ) ＊ ...(33)

[0131] The numerical expression in the above coordinate system A2 is expressed by a superscript (A2). The numerical expression of an arbitrary point D in the coordinate system A2 can be expressed in component notation as follows: D （Ａ２） = (-Sc(N Ｔ２１ P Ｔ (N Ｈ )(D-β)), -Sc(N Ｔ２２ P Ｔ (N Ｈ )(D-β)))...(34)

[0132] α'' （Ａ２） , β'' （Ａ２） , γ′′ （Ａ２） The coordinate value and the angle of view (θ'' αβ , θ′′ βγ , θ′′ γα) has been found, the coordinate value X'' can be calculated by the same method as in Figure 13 to find X'' from the intersection of the circle. （Ａ２） The coordinate value X'' obtained in this way is （Ａ２） The component expression of X'' is as follows: （Ａ２） = ((X'' （Ａ２） ) １ , (X'' （Ａ２） ) ２ ) ... (35)

[0133] From the above results, the numerical expression X in the vehicle-mounted terminal 1A of the terminal position X of the HMD 1B is （Ａ） Here, it should be noted that X' is a point included in both the horizontal and vertical projection planes, depending on how the horizontal and vertical projection planes are selected (Fig. 14). Therefore, the positional relationship between X' and X is N Ｖ If we pay attention to the alignment of the points in the direction, X is a point in the selected vertical projection plane. Therefore, the position of point X'' projected from X onto the vertical projection plane is equal to X. Therefore, X （Ａ） is calculated using the following formula: X （Ａ） =X'' （Ａ） = (X'' （Ａ２） ) １ N Ｔ２１ （Ａ） + (X'' （Ａ２） ) ２ N Ｔ２２ （Ａ） +β （Ａ） ...(36)

[0134] Here, we will explain points to note when estimating the terminal position X of the HMD 1B from the in-vehicle terminal 1A using the theorem of circular angles shown in FIG. 13 . As shown in FIG. 15 , there are cases where the three external field feature points α', β', and γ' and the terminal position X' in the horizontal projection plane of the HMD 1B are located on the same circumference. In this case, the terminal position X' of the HMD 1B cannot be determined using the process shown in FIG. 13 . In this case, another external field feature point that is not located on the same circumference is substituted (substitute external field feature point). The substitution with the substitute external field feature point is achieved by the processor 125A of the in-vehicle terminal 1A automatically setting a fourth point when the in-vehicle terminal 1A executes the process of FIG. 13 using the three external field feature points α', β', and γ' but is unable to estimate the terminal position X'. The same applies to processing in the vertical projection plane. To prepare for such cases, it is desirable to set the number of external field feature points whose positions are to be measured to four or more.

[0135] Below, we will determine the three-dimensional positional relationship between the user information terminal and the other person's information terminal, but in a normal meeting situation, the user information terminal and the other person's information terminal are located approximately on the same plane, so it is sufficient to only determine the positional relationship within the horizontal projection plane.

[0136] Through the processing up to this point, the terminal position X of the HMD 1B and the positions of each external feature point in the coordinate system of the vehicle-mounted terminal 1A are known. Now, if any two points are selected from the external feature points, for example, the external feature point β and the external feature point γ, a direction vector N from the terminal position X to the external feature point β is obtained. １ and a direction vector N from the terminal position X to the external feature point γ. ２ The numerical expression in the coordinate system of the vehicle-mounted terminal 1A is obtained (FIG. 16).

[0137] On the other hand, in the HMD 1B, the position and orientation sensor 135B of the HMD 1B detects the coordinate value of the HMD 1B in the coordinate system of the HMD 1B (i.e., the coordinate value of the terminal position X) and the direction vector N １ and N ２ We know the numerical representation of N １ and N ２ It is assumed that external feature points are selected in advance so that they do not have the same direction.

[0138] Now, the direction vector N in two different external worlds １ and N ２ Since the numerical expressions in the respective terminal world coordinate systems are obtained, the rotation Q is used to convert the numerical expression of the direction vector expressed in the coordinate system of the HMD 1B into the numerical expression of the direction vector expressed in the coordinate system of the vehicle-mounted terminal 1A by the method described in the method (1) for acquiring the positional relationship between the terminals. ＢＡ can be obtained (FIGS. 10 and 11).

[0139] Next, similarly to equation (16), the difference between the coordinate system origins is calculated using information on points whose numerical expressions in the coordinate systems of both terminals are known. ＢＡ is as follows: O ＢＡ =X （Ａ） -Q ＢＡ X （Ｂ） Q ＢＡ ＊ ...(37)

[0140] Here, X （Ａ） , X （Ｂ） are numerical representations of the terminal position X in the coordinate system of the vehicle-mounted terminal 1A and the coordinate system of the HMD 1B, respectively.

[0141] As with the method (1) for obtaining the positional relationship between terminals, the rotation Q ＢＡ and the difference between the origin position O ＢＡ By sharing this information between both terminals, each terminal can obtain the position coordinates of the other terminal in its own coordinate system and convert them into coordinates in its own coordinate system. In other words, both terminals can grasp the positional relationship between the terminals. Alternatively, one terminal may perform the coordinate conversion calculation and share the calculation results.

[0142] (Modification of Method (2) for Acquiring Positional Relationship Between Terminals) In the above, the projection plane is set based on the vertical direction that can be recognized by both terminals, but this is not limited to this as long as it is an external direction that can be recognized by both terminals. For example, the direction of north may be measured by the geomagnetic sensors 117A and 117B and used as the reference direction. Also, for example, the direction of a landmark that is sufficiently far from the terminals may be used as the reference direction. This reference direction vector is referred to as "N" in the above processing.Ｖ " and the same procedure can be performed.

[0143] (Summary of methods (1) and (2) for acquiring the positional relationship between terminals) Here we will summarize the methods for acquiring the positional relationship between two terminals. Hereinafter, the entity that processes data is a processor installed on one of the terminals or the server. The data to be processed is sent from each terminal to the terminal or server that will perform the processing, and the terminal or server that performed the data processing sends the processing results to the terminal or server that will use the processing results. Ultimately, each terminal uses the data processing results to recognize the relative position of the other terminal with respect to its own terminal, and uses this as support information for executing the rendezvous.

[0144] (1) The approximate location of the meeting is recognized by both terminals. This recognition method may involve, for example, sharing the name of the meeting place. The meeting place may be specified by the name of a street, the name of a nearby building or facility, or an address. Alternatively, location recognition information obtained by a common positioning system implemented on both terminals may be used. This common positioning system may be, for example, a positioning method that uses satellites such as GPS. Note that in the present invention, even if the specific positioning methods are different, those that use common location expressions such as latitude, longitude, and altitude are considered to be the same common positioning system.

[0145] (2) Both devices capture images of the outside world near the meeting place (first and second outside images). Because the same area of ​​the outside world needs to be captured, information about the object to be captured may be shared between both devices. The information about the object to be captured may include, for example, the name of the building or facility to be captured, the position of the object to be captured based on a common positioning system, or information about the location and direction of the image to be captured based on a common positioning system.

[0146] (3) Either one of the terminals or the server compares the images captured by both terminals and performs matching of external feature points. Distributed processing is also acceptable.

[0147] (4) In either the terminal or the server, for a pair of matching external world feature points in the set of first external world feature points and external world feature points in the set of second external world feature points, information on the relative positional relationship between the external world feature points and the terminal in the positioning system of each terminal is obtained. Here, "information on the relative positional relationship between the external world feature points and the terminal" refers to "information on the coordinate values ​​of the terminal position and the coordinate values ​​of the external world feature points" or "information on the coordinate values ​​of the terminal position and the direction of the external world feature points as seen from the terminal position." For at least one of the terminals, "information on the relative positional relationship between the external world feature points and the terminal" needs to be "information on the coordinate values ​​of the terminal position and the coordinate values ​​of the external world feature points." Here, "the positioning system of each terminal" refers to a positioning system by which each terminal recognizes its own position and orientation, and more specifically, refers to a measurement method and expression format for recognizing coordinate values ​​and direction vectors in a terminal world coordinate system defined by each terminal.

[0148] The direction of the external feature point as seen from the terminal position in the positioning system of each terminal described above can be calculated from the external world image captured by that terminal, as shown in Fig. 8. The calculation itself can be performed by other terminals or servers if the terminal position coordinates and image field angle information are transmitted together with the external world image information.

[0149] In order to obtain the coordinate values ​​of the external feature points in the positioning system of each terminal described above, it is necessary to measure the distance from the terminal position to the external feature point in addition to information on the direction of the external feature point as seen from the terminal position. Distance measurement methods are broadly divided into methods using ranging sensors 113A, 113B, and 113C, and a moving stereo method using images captured at different shooting locations. While methods using ranging sensors 113A, 113B, and 113C are performed at the terminal, the moving stereo method may transmit image information from different shooting locations together with information on the shooting locations to a partner terminal (another person's information terminal as seen from the user information terminal, or the user information terminal as seen from the other person's information terminal) or a server, and the distance may be calculated at the partner terminal or server. Hereinafter, in distance measurement using the moving stereo method, it is assumed that the executing entity holds the information on the shooting location and image information required for measurement, or acquires that information from the shooting terminal. Furthermore, it is necessary to match external feature points in the images between the multiple images used in the moving stereo method, and this matching can be performed using the same method as the external feature point matching between the images of both terminals.

[0150] The distance between the device and the external world feature point may be measured before performing external world feature point matching. In this case, the device that performed the distance measurement transmits list information that pairs the position information of the external world feature in the external world image with the distance information to the device or server that uses the distance information. Here, the execution of distance measurement may be limited to the external world feature points extracted as external world feature points, or distance measurement to measurable external world points may be performed. In this method of measuring distance before external world feature point matching, there is a possibility that distance information is not available for the matched external world feature point. In this case, the matching may be treated as not being successful, or, as described below, distance measurement of the relevant external world feature point may be performed after external world feature point matching.

[0151] Measurement of the distance between the terminal and the external feature points may be performed after performing external feature point matching. In this case, if the terminal or server performing the distance measurement is not performing the matching process, the terminal or server that performed the matching process transmits information about the external feature points for which matching was achieved to the terminal or server performing the distance measurement. After the distance measurement is performed, the entity performing the distance measurement transmits the distance information to the terminal or server that requires it.

[0152] (5) Using the method (1) or (2) for acquiring the positional relationship between terminals, the terminal or server determines the "rotation and origin difference," which is the coordinate system transformation parameter between the world coordinate systems of both terminals. After the coordinate system transformation parameter is determined, the executing entity for calculating the coordinate system transformation parameter transmits the information to the terminal or server that uses the coordinate system transformation parameter.

[0153] Here, we summarize how to calculate coordinate system transformation parameters in methods (1) and (2) for acquiring the inter-device positional relationship, including both methods. First, a rotation parameter for aligning the orientations of the two device world coordinate systems (first device world coordinate system and second device world coordinate system) is calculated from the direction vector values ​​of two direction vectors defined in the external world in two device world coordinate systems. Next, the difference between the origins of the two device world coordinate systems is calculated from the coordinate values ​​of one position defined in the external world in the two device world coordinate systems. Here, the two direction vectors defined in the external world refer to two direction vectors selected from the direction vector from the device position to the external world feature point position or the direction vector from one external world feature point to another external world feature point. Furthermore, the one position defined in the external world refers to either one of the device positions or one external world feature point position.

[0154] (6) Each terminal transmits its own location information to the other terminal or server that uses this location information in advance or each time it is updated. This location information may be a numerical representation of the terminal's positioning system, or may be converted into a numerical representation of the other terminal's positioning system after the coordinate system transformation parameters are calculated. Alternatively, both the numerical representations of the positioning systems of both terminals may be transmitted. Each terminal or server executes a process to support rendezvous based on the terminal's location information. Examples of rendezvous support processes include superimposing the location of the other terminal on an external image, or using the information to control the driving of a vehicle such as a car to guide the other terminal to its location.

[0155] (Modification (1) common to methods (1) and (2) for acquiring positional relationship between terminals) In the method of the present invention, a direction vector to an external feature point is acquired from an external image captured by the terminal, so it is preferable that distortion of the optical system is corrected and distortion of the captured image is minimized. Therefore, if the outer cameras 111A, 111B, and 111C of the terminal have a low-distortion shooting mode, the external image for acquiring the relative position may be automatically captured in the low-distortion shooting mode. Figure 17 shows an example in which a message for switching to low-distortion shooting mode is superimposed on the captured image preview screen. Switching to low-distortion shooting mode can improve the accuracy of estimating the relative position.

[0156] (Variation (2) common to methods (1) and (2) for acquiring positional relationship between terminals) In the method of the present invention, it is essential that both terminals capture images of the same area of ​​the outside world, so the image used to acquire the relative position may be wide-angle image. Therefore, if the outer cameras 111A, 111B, and 111C of the terminals have a wide-angle mode, wide-angle image may be automatically captured when capturing image for acquiring the relative position. Figure 18 shows an example in which a message for switching to wide-angle capture mode is superimposed on the captured image preview screen. Switching to wide-angle capture mode increases the certainty of capturing images of the same area of ​​the outside world, reducing the risk of taking additional images.

[0157] (Modification (3) common to methods (1) and (2) for acquiring the positional relationship between terminals) As with modification (2) described above, a shooting guide may be displayed as a method for increasing the certainty that both terminals will shoot the same area of ​​the outside world. The shooting guide here is to display the image shot by the other terminal on the shot image preview screen of the terminal itself when the other terminal shoots an image of the outside world first. An example of such a display is shown in FIG. 19. By displaying the image shot by the other terminal and providing the user with guidance on the shooting direction, the certainty that the same area of ​​the outside world will be shot increases, and the risk of taking additional images can be reduced.

[0158] (Modification (4) common to methods (1) and (2) for acquiring positional relationship between terminals) In addition to the above-described modification (3), another shooting guide may be displayed as a method for increasing the certainty that the same area of ​​the outside world is photographed on both terminals. The shooting guide here is to display an AR object or the like indicating the target shooting point on the shooting video preview screen of the terminal itself, based on target shooting point information based on a common positioning system such as GPS that allows both terminals to share position recognition. An example of such a display is shown in FIG. 20. By displaying an AR object or the like indicating the target shooting point on the shooting video preview screen of the terminal itself, the certainty that the same area of ​​the outside world is photographed is increased, and the risk of taking additional photographs can be reduced.

[0159] (Modification (5) common to methods (1) and (2) for acquiring positional relationship between terminals) In the above process, three matched external feature points are used (one or two external feature points are used when a specific direction in real space is used) to calculate the rotation and origin difference, which are the transformation parameters of the coordinate systems of both terminals. However, more external feature points may be used. That is, multiple coordinate system transformation parameters may be obtained from multiple sets of three external feature points, and their average value may be used. When a specific direction in real space is used, the number of external feature points is determined to be the number required for that case. In this case, when looking at the distribution of the coordinate system transformation parameters, any data that deviates significantly from the average value may be considered anomalous data and excluded from the data for calculating the average value, and the calculation may be repeated. Well-known methods such as the Smirnoff-Grubbs test may be used to determine anomalous data. For example, if the standard deviation of the distribution is σ, data that deviates by 3σ or more from the average value may be determined to be anomalous data.

[0160] The process of calculating the average value may be performed on the coordinate values ​​in the terminal's own world coordinate system of the location of the other terminal estimated using the coordinate system transformation parameters. In this case, when examining the distribution of estimated coordinate values, any values ​​that deviate significantly from the average value may be regarded as abnormal data and excluded from the data for calculating the average value, and the calculation may be retried. The coordinate system transformation parameters corresponding to these abnormal estimated coordinate values ​​are also recognized as abnormal data.

[0161] Among the three external field feature points corresponding to coordinate system transformation parameters determined as abnormal data using the above-described method, there is a possibility that at least one is a false external field feature point that has been accidentally matched between two images. Therefore, external field feature points belonging to a triplet of external field feature points corresponding to abnormal data equal to or greater than the set number of pairs may be determined as false external field feature points, and the matching may not be established. In other words, this is additional screening of external field feature point matching based on the calculation results of coordinate system transformation parameters.

[0162] After the above-described additional screening of external boundary feature points is performed, a plurality of coordinate system transformation parameters may be newly calculated from a plurality of triplet sets of external boundary feature points, and the process of calculating the average value thereof may be repeated.

[0163] By performing the processing of this modified example, it is possible to improve the accuracy of estimating the relative position.

[0164] (Variation (6) Common to Methods (1) and (2) for Acquiring Positional Relationship Between Terminals) If the number of matched external world feature points is less than a set number, or if the variation in the transformation parameters of the coordinate systems of both terminals calculated using the matched external world feature points is greater than a set amount, it is possible that the locations captured by both terminals are different, or that the captured ranges of the images have little overlap. In this case, if additional capture is possible, a message urging the users of both terminals to take additional capture may be displayed by visual display or audio notification. This increases the likelihood that the same area of ​​the external world will be captured if additional capture is performed ( FIG. 21A ). Furthermore, if it is determined that a terminal has moved away from the capture location, the terminal may stop issuing the above-mentioned message and notify the other terminal that it is unable to take additional capture, prompting the user of the other terminal to take additional capture ( FIG. 21B ). This improves the effectiveness of the message.

[0165] Furthermore, when the moving terminal (vehicle-mounted terminal 1A) approaches the meeting place, only the vehicle-mounted terminal 1A may additionally capture an external image of the vicinity of the meeting place. This changes the image on the vehicle-mounted terminal 1A side to an image captured from a location closer to the location captured by the HMD 1B. This increases the reliability of matching of external world feature points, increases the number of matching external world feature points, and improves the accuracy of estimating the relative positions of the two terminals.

[0166] (Information Display) FIG. 22 is a diagram showing an example of information display on the user information terminal (vehicle-mounted terminal 1A) based on terminal position estimation for meeting support.

[0167] 22 , an outside image 1200 captured by the outer camera 111A is displayed on the display 119A of the in-vehicle terminal 1A. Furthermore, virtual objects 1210 and 1211 indicating the terminal position X of the HMD 1B calculated by the processor 125A of the in-vehicle terminal 1A are displayed superimposed on the outside image 1200 of the display 119A. The terminal position X corresponds to the estimated position of the passenger.

[0168] FIG. 23 is a diagram showing an example of information display on the other person's information terminal (HMD 1B) based on terminal position estimation for meeting support.

[0169] 23 , an outside world image 1300 captured by the outer camera 111B is displayed on the display 119B of the HMD 1B. Furthermore, the HMD 1B receives information indicating the relative positional relationship between the in-vehicle terminal 1A and the terminal position X of the HMD 1B, calculated by the processor 125A of the in-vehicle terminal 1A. The processor 125B of the HMD 1B then calculates the terminal position of the in-vehicle terminal 1A relative to the HMD 1B using the information indicating the relative positional relationship, and displays virtual objects 1310 and 1311 indicating the position of the in-vehicle terminal 1A superimposed on the outside world image 1300 on the display 119B. The objects 1310 and 1311 are virtual objects indicating the estimated position of a taxi.

[0170] This embodiment makes it possible to provide highly accurate rendezvous guidance.

[0171] The information display in the automobile using the in-vehicle terminal 1A may be a HUD (Head Up Display).

[0172] If the estimated location of the other party's terminal is not included in the range shown on the display, the direction to be displayed may be guided by an arrow display, audio, etc. The direction of the meeting party may be presented by sound image localization using 3D audio.

[0173] Furthermore, the information on the location of the other party's terminal may be used not only to confirm the other party but also to control the speed of the car so that the car stops quickly and without abrupt stops in front of the other party. The speed control may be used to recommend a speed to the driver, or for speed control in a driving support system or an automatic driving system.

[0174] 24 is a flowchart showing the processing flow of the meeting support system 100 according to this embodiment. Although the server 2 is not explicitly shown, some of the processing may be performed by the server 2 to distribute the processing load.

[0175] First, the in-vehicle terminal 1A and the HMD 1B accept the setting of a meeting place by the user and start processing. A specific example of use is when a user of the HMD 1B requests a taxi to pick them up and sets the meeting place on the in-vehicle terminal 1A installed in the requested taxi. After setting the meeting place, the in-vehicle terminal 1A and the HMD 1B, for example, capture external world images near the meeting place using the outer cameras 111A and 111B, acquire terminal position coordinates at the time of capture in the respective terminal world coordinate systems, and measure the distance from the terminal position to an external world feature point (S01A, S01B). The distance measurement may be performed using distance measurement sensors 113A and 113B such as LiDAR or TOF, or a moving stereo method using multiple images captured at different positions. Note that if the method (2) for acquiring the positional relationship between terminals is used, distance measurement can be omitted in one of the terminals, so distance measurement is not performed in the terminal that omits distance measurement. Alternatively, distance measurement at this stage may not be necessary in the terminal that measures distances to external feature points after feature point matching.

[0176] Next, the in-vehicle terminal 1A and the HMD 1B share the captured external world images via the network communication devices 120A and 120B (S02A, S02B). Hereinafter, the transmission and sharing of information will be assumed to be performed via the network communication devices 120A, 120B, and 220, and an explicit description thereof will be omitted. Here, the external world image data may be aggregated only in the terminal that performs external world feature point matching or in the server 2.

[0177] Next, the processors 125A and 125B of the in-vehicle terminal 1A and the HMD 1B compare the images captured by both terminals, match the external feature points, and share the matching results (S03A, S03B). Here, the matching process may be executed by the processor of either terminal, or by the processor 225 of the server 2. Hereinafter, in text expressions in which the in-vehicle terminal 1A, the HMD 1B, or the server 2 is the processing subject, it is assumed that the processing subjects at the hardware component level shown in Figures 2 and 3 are the processors 125A, 125B, and 225.

[0178] Here, the vehicle-mounted terminal 1A and the HMD 1B determine whether the number of matching external feature points is equal to or greater than a predetermined number (S04A, S04B).

[0179] If the determinations in S04A and S04B are negative (S04A, S04B: No), the in-vehicle terminal 1A and the HMD 1B return to steps S01A and S01B. If one of the terminals, for example, the HMD 1B, has moved from the shooting point to a distant location and is unable to take additional images, the HMD 1B skips step S01B and repeats the process from S02B using the previous image.

[0180] If the determinations in S04A and S04B are positive (S04A and S04B: Yes), the in-vehicle terminal 1A and the HMD 1B perform distance measurement for the matched external field feature points (S05A and S05B). If distance measurement has been performed in the steps of S01A and S01B, distance measurement at this step may be omitted. Furthermore, distance measurement is not performed in terminals that omit distance measurement.

[0181] In the distance measurement in steps S05A and S05B, if the terminal has not moved from the shooting point of the video used for matching, distance measurement is possible using distance measurement sensors 113A and 113B such as LiDAR or TOF. If the terminal moves and distance measurement of the external world feature point by distance measurement sensors 113A and 113B becomes impossible, the in-vehicle terminal 1A and HMD 1B capture multiple images in advance with the outer cameras 111A and 111B by shifting the shooting point within a range where the same external world area can be captured, and measure the distance from the shooting point to the external world feature point by the moving stereo method.

[0182] Next, the in-vehicle terminal 1A and the HMD 1B calculate the coordinates of the external world feature points in their respective terminal world coordinate systems using the results of the distance measurements from the terminal position to the external world feature points performed up to this stage, and share the terminal position coordinates and the external world feature point coordinates (S06A, S06B). Note that the calculation of the external world feature point coordinates can be performed after the distance measurement, so it may be performed before this stage. Furthermore, the in-vehicle terminal 1A and the HMD 1B may transmit the results of the distance measurement to another terminal or the server 2 to have the external world feature point coordinates calculated, and share the results.

[0183] Next, the in-vehicle terminal 1A and the HMD 1B use the terminal position coordinates and the external world feature point coordinates to calculate and share transformation parameters between the world coordinate systems of both terminals (S07A, S07B). Note that the in-vehicle terminal 1A and the HMD 1B may transmit information on the terminal position coordinates and the external world feature point coordinates to a terminal or server 2 that performs calculation processing of the terminal world coordinate system transformation parameters, have the transformation parameters between the world coordinate systems calculated, and share the results. The process of having the other party's terminal or server 2 perform the calculation may be performed consecutively from the stages of steps S06A and S06B.

[0184] Next, the in-vehicle terminal 1A and the HMD 1B transmit their own current position information to each other terminal, and use the conversion parameters between the world coordinate systems to grasp the position of the other terminal in the world coordinate system (S08A, S08B). This calculation of the positions of each terminal may also be performed in one of the terminals or in the server 2.

[0185] At this point, the vehicle-mounted terminal 1A and the HMD 1B determine whether to update the external world image (S09A, S09B). This determination is made based on whether the variation in the estimation result of the partner terminal position exceeds a certain limit and on how close the moving terminal (vehicle-mounted terminal 1A) has come to the meeting place.

[0186] The variance of the estimation results is determined by, for example, updating the external world image when the standard deviation of the estimated position distribution is equal to or greater than a predetermined value. The estimated positions are calculated as three-dimensional or two-dimensional values, and the standard deviation mentioned above refers to the standard deviation of the distribution of distances of each estimated position from the average position.

[0187] The degree of approach of the vehicle-mounted terminal 1A to the meeting place is determined by the distance between the terminal position of the vehicle-mounted terminal 1A and the estimated terminal position of the HMD 1B. The image is updated when a predetermined threshold value for the estimated inter-terminal distance is reached. Multiple threshold values ​​may be defined. A standard deviation of the target estimated position is determined for each threshold value, and it is determined that the image should be updated if the estimated position is not equal to or less than that standard deviation.

[0188] If the determinations in S09A and S09B are positive (S09A, S09B: Yes), the in-vehicle terminal 1A and the HMD 1B return to the processing of steps S01A and S01B. However, once the determinations in S09A and S09B become negative (S09A, S09B: No) and the output of meeting support information (S10A, S10B) has started, the in-vehicle terminal 1A and the HMD 1B continue the processing of steps S08A, S08B, S10A, and S10B in parallel using the existing coordinate transformation parameters until the external world image is updated and new coordinate transformation parameters are obtained according to the flow after returning to the processing of steps S01A and S01B.

[0189] If the determinations in S09A and S09B are negative (S09A, S09B: No), the in-vehicle terminal 1A and the HMD 1B output meeting support information (S10A, S10B). Note that if the processing of steps S08A, S08B, S10A, and S10B has been continued using existing coordinate transformation parameters, the in-vehicle terminal 1A and the HMD 1B change to newly obtained coordinate transformation parameters and continue the processing of steps S08A, S08B, S10A, and S10B until the termination condition (S11A, S11B) is satisfied.

[0190] In steps S10A and S10B, the in-vehicle terminal 1A and the HMD 1B each output information for meeting support based on the estimated position of the other terminal. Note that in the in-vehicle terminal 1A, the information output for meeting support may be for automatic driving control. In addition, in the HMD 1B, if the user simply waits at the meeting place, there is no need to use information for meeting support, such as displaying the estimated position of the other terminal.

[0191] Next, if the termination condition is satisfied (S11A, S11B: Yes), the in-vehicle terminal 1A and the HMD 1B terminate the process, and if the termination condition is not satisfied (S11A, S11B: No), the process returns to steps S01A and S01B. The termination condition for the in-vehicle terminal 1A may be determined to be satisfied, for example, when the in-vehicle terminal 1A arrives at the meeting place, picks up a passenger, and turns on the taxi meter. The termination condition for the HMD 1B may be determined to be satisfied when the distance between the terminal positions of the in-vehicle terminal 1A and the HMD 1B is less than a distance threshold, for example, 1 meter.

[0192] In this embodiment, both devices capture images of the external world around the meeting place at the time of the meeting, and the location of the other device in its own device's world coordinate system is estimated by comparing the numerical representations of external feature points in the images in each device's world coordinate system. This makes it possible to easily and accurately estimate the location of the other device without being affected by position measurement errors in a general-purpose common positioning system, such as reflection of satellite radio waves by buildings, and without the need to prepare a huge amount of external world images of the area around potential meeting places in advance. Furthermore, compared to using library images, this invention uses images from the time of the meeting, which has the advantage of not being affected by changes in the image that may occur between the time the library images were captured and the time of use.

[0193] <Second Embodiment> In a second embodiment, in order to perform more precise speed control of the vehicle, some or all of the external field feature points may be set on the body of the vehicle on which the in-vehicle terminal 1A is mounted. Fig. 25 is a diagram showing an example of setting the external field feature points in the second embodiment. In this case, the image of the vehicle used for matching the external field feature points is one that has been captured in advance. If it is possible to identify which part of the vehicle body is involved, the coordinate values ​​in the terminal local coordinate system can be determined, and by referring to the time of capturing the image on the HMD 1B, the coordinate values ​​in the terminal world coordinate system can be calculated.

[0194] 25, in a situation where a car is approaching the vicinity of the meeting point, the external world feature points set on the car body will be closer to the HMD 1B than the external world feature points set on a building, etc., and furthermore, for the in-vehicle terminal 1A, these are external world feature points whose position coordinates are known accurately, so the accuracy and precision of position estimation will be improved more by setting external world feature points on the car body than by setting external world feature points on a building, etc. In this case, in order to confirm that the car is the one to be met, an identification number or the like displayed on the car body that can be seen in the video may be recognized.

[0195] <Third embodiment> In a third embodiment, the information display (FIG. 22) of the in-vehicle terminal 1A may also be displayed on the HMD 1B, and the passenger may check whether the passenger has been correctly identified by estimating the position of the destination point. If the passenger has been incorrectly identified, the HMD 1B may transmit a message to that effect to the in-vehicle terminal 1A. The passenger may manually input the determination as to whether the passenger has been correctly identified into the HMD 1B, or the processor 125B of the HMD 1B may automatically determine whether the passenger has been correctly identified.

[0196] Furthermore, if the passenger is incorrectly captured in the image sent from the vehicle-mounted terminal 1A, the true passenger position can be indicated to the vehicle-mounted terminal 1A based on the position in the image, thereby correcting the passenger's identification. If the passenger is not captured in the image sent from the vehicle-mounted terminal 1A, the HMD 1B may notify the vehicle-mounted terminal 1A of the direction in which the true passenger position deviates from the image. Furthermore, if the estimation of the passenger position is incorrect, both terminals may re-capture the external world image around the target point, and the relative position may be re-estimated.

[0197] After the passenger to be met is confirmed on the video, the speed of the vehicle may be controlled so as to stop the vehicle at a road position nearest to the passenger.

[0198] According to this embodiment, the destination can be confirmed by other passengers using the information terminal. In addition, by diverting the technology to speed control, it is expected that the accuracy of stopping an autonomous taxi vehicle in front of a passenger can be improved.

[0199] Fourth Embodiment The fourth embodiment is a modified example in which the photographing point on the HMD 1B side, which is the other person's information terminal, is different from the meeting place (FIG. 26).

[0200] In cases where the meeting place is a narrow road off a main road and video captured from the meeting place is not suitable for meeting support, for example, video captured from a position other than the meeting place may be sent to the vehicle-mounted terminal 1A as a comparison video. In this case, since the HMD 1B can grasp its own position in the world coordinate system using the position and orientation sensor 135B, it transmits the coordinate values ​​of the meeting place (target point) in addition to the coordinate values ​​of the shooting point to the vehicle-mounted terminal 1A. The vehicle-mounted terminal 1A can grasp the meeting place by converting the coordinate values ​​of the meeting place in the terminal world coordinate system of HMD 1B into coordinate values ​​in the terminal world coordinate system of the vehicle-mounted terminal 1A using the transformation parameters between coordinate systems obtained by the method of embodiment 1.

[0201] Furthermore, the user of the HMD 1B may be waiting for a taxi inside a building. In this case, the HMD 1B may transmit the coordinate values ​​of a point in front of the building as the meeting place to the vehicle-mounted terminal 1A, and may transmit an image of the building where the user of the HMD 1B is waiting to the vehicle-mounted terminal 1A for confirmation of the meeting place.

[0202] According to this embodiment, video is captured from a shooting position that is different from the meeting place, data necessary for creating external world feature point information is measured, and then the vehicle can move to the original meeting place. Therefore, even if the meeting place is difficult to find because it faces a narrow alley, by setting up a shooting position on a main street, it becomes easier to support a taxi in approaching the meeting place. It is also possible to wait for a taxi at a location away from the meeting place. Note that the external world feature point information in this case is information that directly indicates the relative positional relationship between the external world feature points and the video shooting position, but it can also be said to be information that indirectly indicates the relative positional relationship between the external world feature points and the information terminal.

[0203] Fifth Embodiment In a fifth embodiment, images of external feature points captured by passengers may be provided to a taxi company to be included in a video library. In this case, passengers may be given an incentive, such as a discounted fare. Because the state of the external world may change over time, it is desirable to keep updating the video library with as much new video as possible. This allows the vehicle-mounted terminal 1A to perform location estimation based on the latest situation when using library video, thereby improving the accuracy of location estimation.

[0204] Sixth Embodiment This embodiment is a case where two users are both moving around to find each other. For example, a meeting is planned in a place where GPS signals cannot reach, such as a shopping mall, with only a rough location, such as a certain floor, being decided. In this embodiment, the process is described in the form of a meeting between users of a first information terminal and a second information terminal, but in this embodiment, it is assumed that the first information terminal and the second information terminal may be mobile terminals such as HMDs or smartphones.

[0205] FIG. 27 is a diagram illustrating the movement of the information terminal in the sixth embodiment.

[0206] A use case of this embodiment will be described with reference to Fig. 27. Fig. 27 shows the trajectories of a user of a first information terminal and a user of a second information terminal who move around an area of ​​a meeting place (corresponding to the "meeting place area") in a building in search of the other person. In the figure, the trajectories approach each other at a location indicated as the "trajectory approaching portion." The meeting support method of this embodiment uses external feature points (referred to as "external feature points") near this location to grasp the relative positional relationship between the two terminals.

[0207] FIG. 28 is a diagram for explaining the video matching process in the sixth embodiment, and shows the concept of the process for grasping the current relative positional relationship between both terminals.

[0208] First, both devices acquire images of the surrounding environment while moving, and simultaneously record changes in their position and orientation. This information on the images of the surrounding environment, the device's position, and the device's orientation is then shared between the two devices. Alternatively, data can be aggregated on one device or on a server to perform relative position calculations, and the calculation results can be notified to the device that is not performing the calculations.

[0209] To calculate the relative position, first, past images of the outside world taken by both terminals are searched for images of the same location. This location is the trajectory approach section in Figure 27. From the images of the outside world taken by both terminals at the same location in this trajectory approach section, coordinate transformation parameters between the coordinate systems of both terminals are calculated using the method described above. Once the coordinate transformation parameters are known, the current coordinate values ​​of the other terminal's position can be converted to the coordinate values ​​of the own terminal, thereby determining the positional relationship between the own terminal and the other terminal.

[0210] Here, to determine the relationship between the coordinate systems of the two devices, the coordinates of external feature points in the external world images are used, so it is necessary to measure the distance from the device to each external feature point on at least one device. This measurement can be performed using a moving stereo technique using multiple images taken from different terminal positions near the trajectory approach point. By using this method, it becomes possible to select feature points that appear in both images captured by both devices later and measure only the distance to those feature points from the previous and next images, without having to determine the external feature points to be used in advance. Point cloud data can also be obtained by measuring the distance to each point in the image using LiDAR or the like.

[0211] Here, the trajectory approach section has been described as a point that both terminals have passed through in the past, but if the video search speed is fast enough, a terminal that arrives at the trajectory approach section later can search the past video of the other terminal on the spot and recognize that the location at the trajectory approach section is the trajectory approach section.

[0212] In the example of FIG. 28, the number of matching external feature points is checked between frame tAm (m=0, 1, 2, . . . ) constituting the external image captured by the first information terminal and frame tBm (m=0, 1, 2, . . . ) constituting the external image captured by the second information terminal. １Once the frame is obtained, the number of matching external feature points between the frame and the already obtained frame tBm (m=0, 1, ..., m1) is checked. １ Once a frame tAm (m=0, 1, ..., m1) is obtained, the number of matching external feature points between the frame tAm and the frame tAm (m=0, 1, ..., m1) already obtained at the time the frame is obtained is checked. Note that, although the shooting time is used as the video index in FIG. 28, this is not limiting. Furthermore, the video shooting times of the first and second information terminals do not need to be the same.

[0213] During this procedure, a pair of frames (hereinafter sometimes referred to as a matching pair) that has a predetermined number or more of matching external field feature points is determined to be an image pair of the trajectory approach portion. In the example of FIG. 28 , a pair of frames tA1 and tB2 is selected as the trajectory approach portion image pair. That is, the position of the first information terminal when capturing the first external field image, which is one of the trajectory approach portion image pair, and the position of the second information terminal when capturing the second external field image, which is the other of the trajectory approach portion image pair, are determined to be the position of the trajectory approach portion. This method uses the first matching pair found for subsequent calculation of coordinate transformation parameters, but there may be multiple matching pairs. After several matching pairs are found, the matching pair with the largest number of matching external field feature points may be used for subsequent calculation of coordinate transformation parameters. The greater the number of matching external field feature points, the more likely it is that the accuracy of the coordinate transformation parameters will be improved. Note that, as a method for matching external field feature points, the method described in the first embodiment, for example, is used. When referring to external feature points in an image, the external feature points in the first external image are referred to as first external feature points, and the external feature points in the second external image are referred to as second external feature points.

[0214] In the above process, it is important to continuously measure changes in the terminal's position and orientation while capturing surrounding images. Here, we will show an example of measuring changes in the terminal's position and orientation using an acceleration sensor and a gyro sensor as position and orientation sensors.

[0215] FIG. 29 is a diagram illustrating the relationship between the terminal local coordinate system and the terminal world coordinate system.

[0216] After entering the waiting area, the position and orientation of each terminal are measured in their respective terminal positioning systems. Here, the position and orientation of the terminal with respect to the terminal world coordinate system can be expressed by a relational expression for coordinate transformation between the terminal local coordinate system and the terminal world coordinate system (see FIG. 29, Equation (11)).

[0217] The coordinate value O of the origin of this terminal's local coordinate system in the terminal's world coordinate system LW and the rotation Q from the reference point in the terminal's local coordinate system in the terminal's world coordinate system LW are the position and orientation of the terminal in the terminal world coordinate system. At the reference point in time, the orientation of the coordinate axes of the terminal local coordinate system is assumed to match the orientation of the coordinate axes of the terminal world coordinate system. Examples of reference points in time include the time when the terminal world coordinate system is initially set when the terminal is turned on, and the time when the terminal world coordinate system is reset. Each terminal measures its own position and orientation using its own terminal positioning method. This positioning is called "autonomous positioning." Here, an example of "autonomous positioning" using acceleration sensors 115A, 115B and gyro sensors 116A, 116B will be described as an example. The content is common to the first information terminal and the second information terminal.

[0218] First, the time t ０ The initial position at O LW０ , the initial orientation is Q LW０ After the initial state, the state is updated at a constant minute time interval Δt. First, the recurrence formula for the change in orientation is shown. Let ω be the numerical expression in the terminal local coordinate system of the angular velocity vector that is the output of the gyro sensors 116A and 116B at time t. Ｌ At this time, the orientation of the terminal at time Δt is ω Ｌ The angle |ω with the direction of (t) as the axis Ｌ (t) | Rotate by Δt. Let the quaternion that represents this rotation be Δq Ｌ Here, |q| is the norm of the quaternion q. This Δq Ｌ is expressed as follows: Ｌ = cos(|ω Ｌ (t) |Δt / 2) +SIN(|ω Ｌ (t) | Δt / 2) [ω Ｌ(t)] =1+(|ω Ｌ (t) | Δt / 2) [ω Ｌ (t)]+O((Δt) ２ ) = 1 + ω Ｌ (t)Δt / 2+O((Δt) ２ ) ... (38)

[0219] Here, O((Δt) ２ ) is (Δt) ２ This means that the quantity is on the order of Δq. Ｌ The numerical expression in the terminal world coordinate system is Δq W Then, the rotation Q LW The recurrence formula for is as follows:

[0220] Q LW (t + Δt) = Δq W Q LW (t) = Q LW (t) Δq Ｌ Q LW (t) ＊ Q LW (t) = Q LW (t) + Q LW (t)ω Ｌ (t)Δt / 2+O((Δt) ２ ) ... (39)

[0221] Rotation Q LW Since Q must be normalized, in actual processing, the calculation result of the right side of equation (39) may be normalized and used as the value at the next time (t+Δt) to prevent error propagation. Specifically, the calculation is as follows: Q LW (t + Δt) = [Q LW (t) + Q LW (t)ω Ｌ (t)Δt / 2]...(40)

[0222] Next, the recurrence formula for the position change is the acceleration a Ｌ This acceleration is also expressed in the terminal local coordinate system, so the expression in the terminal world coordinate system is a Ｗ When converted to (t), the following equation is obtained: Ｗ (t) = Q LW (t) a Ｌ (t)Q LW(t) ＊ ...(41)

[0223] Here, the outputs of the acceleration sensors 115A and 115B include the gravitational acceleration g in addition to the acceleration due to the change in the position of the terminal. Ｗ This gravitational acceleration g Ｗ can be obtained from the outputs of the acceleration sensors 115A and 115B when the terminal is stationary. The velocity of the terminal in the terminal world coordinate system is V Ｗ (t), the gravitational acceleration g Ｗ By correcting the component of V Ｗ The recurrence formula for (t) can be written as follows. The initial value of the terminal velocity is set to 0 when the terminal is stationary. V Ｗ (t+Δt)=V Ｗ (t) + (a Ｗ (t)-g Ｗ ) Δt (42)

[0224] V calculated in this way Ｗ Using (t), O LW The recurrence formula for (t) can be written as follows: LW (t + Δt) = O LW (t) + V Ｗ (t)Δt...(43)

[0225] In this way, the position and orientation of the terminal can be determined by positioning using acceleration sensors 115A and 115B and gyro sensors 116A and 116B. Note that in equations (42) and (43), too, the calculation results on the right-hand sides may be normalized and substituted for the variables on the left-hand sides in order to prevent propagation of errors.

[0226] To calculate the positional relationship between devices, the coordinate values ​​of each device in the device's world coordinate system are required. L0 When this is set, the value R in the terminal world coordinate system Ｗ0 is expressed as follows: R Ｗ0 =Q LW (t)R L0 Q LW (t) ＊ +O LW (t) ... (44) Unless there are special circumstances, R L0Setting it to 0 will save you the trouble of doing the calculations.

[0227] (Video Matching Method) Videos captured by a first information terminal and a second information terminal are matched based on the number of external world feature points that match between the videos. Specifically, each time a new video is captured while moving, the external world feature points are matched with video previously captured by the other terminal. Video pairs that have already undergone matching processing are excluded. Then, when a video pair that exceeds a predetermined number of matches is found, that video pair is used to find coordinate transformation parameters between the terminals using the method of the first embodiment.

[0228] Once the coordinate transformation parameters between the terminals are calculated, the movement route of the other terminal is known, so while moving towards the other terminal, additional images can be taken at the same locations as those taken by the other terminal, and new coordinate transformation parameters can be calculated to improve the accuracy of the coordinate transformation parameters. Specifically, the coordinate transformation parameters {Q ＢＡ , O ＢＡ} are obtained, so that every time new coordinate transformation parameters are obtained, the average value of the previously obtained coordinate transformation parameters is used.

[0229] The video matching and external feature point matching processes may be performed on both devices, on one device, or on the server. Each device transmits the video data it has captured to the device performing the matching process. At that time, the device transmits the coordinate values ​​of the video captured at the time of capture along with the external feature point information.

[0230] (Guidance Display) Fig. 30 is a diagram illustrating an example of guidance display in the sixth embodiment. Fig. 31 is a diagram illustrating another example of guidance display in the sixth embodiment.

[0231] After the location of the other terminal is determined, a display is made to guide the user to the estimated location of the other terminal. Since the movement routes of both terminals from the trajectory approach point are known, a guided route from the user's terminal to the other terminal along those movement routes can be displayed. An example of this guided route superimposed on the external world image as a virtual object is shown in Figure 30. This guided route follows the route taken by both terminals, so it is a route that the user can take.

[0232] In the present invention, when the HMD is a see-through type, the "external world image" is interpreted as the "visual image of the external world" that is visually recognized by the user.

[0233] Another example of guidance display is shown in Figure 31, which shows the direction and location of the other party's terminal. If the structure of the meeting area is known, this display method is simple because if the location of the other party's terminal is known, the shortest route may be found regardless of the route traveled up to that point.

[0234] (Flow Diagram) FIG. 32 is a flowchart showing the flow of processing in the sixth embodiment.

[0235] This flow chart is basically the same as the flowchart of the first embodiment (FIG. 24). The first information terminal corresponds to the user information terminal of the first embodiment, and the second information terminal corresponds to the other person's information terminal of the first embodiment. The correspondence may be interpreted as being reversed. Although the server 2 is not explicitly shown in FIG. 32, some of the processing may be performed by the server 2 to distribute the processing load.

[0236] The difference between the flow of this embodiment and that of the first embodiment is that in the first embodiment, matching of external world feature points is performed between external world images near the meeting place, whereas in this embodiment, matching of external world feature points is performed between external world images of multiple locations captured by both terminals while they are moving.

[0237] Specifically, steps S100A, S100B, S101A, and S101B are steps that differ from Fig. 24. The other steps are the same as those in Fig. 24, so the same numbers are used and the description will be omitted.

[0238] In steps 100A and 100B, matching of external feature points is performed between the video captured by the own terminal and the video of the moving route captured by the other terminal.

[0239] In steps 101A and 101B, it is determined whether or not there are a predetermined number of pairs for which matching has been achieved among the video pairs for which matching processing has been performed.

[0240] If the determination in S101A and S101B is negative (S101A, S101B: No), the first information terminal and the second information terminal return to steps S01A and S01B, and capture an external image of the next location.

[0241] If the determinations in S101A and S101B are positive (S101A and S101B: Yes), the first information terminal and the second information terminal perform distance measurement for matching external feature points in video pairs for which a predetermined number of matches or more have been established (S05A and S05B). If distance measurement has been performed in steps S01A and S01B, distance measurement at this step may be omitted. Furthermore, distance measurement is not performed in terminals that omit distance measurement.

[0242] In the distance measurement in steps S05A and S05B, if the terminal has not moved from the shooting point of the video used for matching, distance measurement is possible using ranging sensors 113A and 113B such as LiDAR or TOF. If the terminal moves and distance measurement of the external world feature point using ranging sensors 113A and 113B becomes impossible, the first information terminal and the second information terminal capture multiple images in advance with outer cameras 111A and 111B by shifting the shooting point within a range where the same external world area can be captured, and measure the distance from the shooting point to the external world feature point using a moving stereo method.

[0243] (Effects of the Present Embodiment) In the present embodiment, both devices capture images of the external world within the meeting area to be used at the time of the meeting, and compare information on external feature points in the images, expressed numerically in the respective device world coordinate systems, to estimate the location of the other device in its own device world coordinate system. This enables the location of the other device to be easily and accurately estimated, even in situations where a general-purpose common positioning system (equivalent to a general-purpose positioning system) such as GPS cannot be used, i.e., outside the general-purpose positioning system area, and without the need to prepare a large amount of external world images within the area to be used as the meeting place in advance. Furthermore, compared to using library images, the present invention uses images from the time of the meeting, which has the advantage of not being affected by changes in the images that may occur between the time the library images were captured and the time of use. Furthermore, in the present embodiment, even if the user moves within the meeting area, past images captured during movement are also referenced, allowing the meeting to be carried out without specifying a specific point within the area.

[0244] (Variation 1) In order to obtain accurate coordinate transformation parameters, the more successfully matched external world feature points are, the better. Therefore, it is desirable that the external world ranges captured by both terminals overlap as much as possible and capture a range in which many distinctive external world points are distributed. For this reason, the terminals determine the quality of the captured range by looking at the distribution of successfully matched external world feature points within the video. When this distribution is in the center of the captured video, it is highly likely that the external world ranges captured by both terminals overlap greatly and capture an external world subject range in which many distinctive external world points that are easy to match are distributed. Therefore, the captured range is updated so that the center of the distribution of matched external world feature points is as close to the center of the captured range as possible. This makes it possible to achieve more accurate position estimation of the other terminal.

[0245] In this embodiment, matching of external feature points is performed between a pair of images taken by the own terminal and the other terminal at a certain location, but the shooting range can be corrected as long as the user does not move too far from the shooting location. This method can also be used in the first embodiment.

[0246] An example of a method for updating the shooting range will be described with reference to Fig. 33. Fig. 33 is a diagram illustrating processing in Modification 1 of the sixth embodiment.

[0247] In the example of FIG. 33 , the distribution of external world feature points that match the original shooting range with the video captured by the other device is displayed as a superimposed virtual object, but this distribution is biased toward the upper right of the shooting range. In this case, the shooting range is updated toward the upper right and new video is captured so that the center of the external world feature point distribution is closer to the center of the shooting range. Alternatively, to include the target area, shooting may be performed in wide-angle mode, or a camera may be used that is pointed in a direction that includes the target area. This updating of the shooting range may be performed automatically by the device, or a process may be performed in which the user is guided in the shooting direction and prompted to reshoot. Examples of displaying guidance for the shooting direction include the display of the matched external world feature point distribution itself in the original captured video, or the center point of the distribution, superimposed on a virtual object. Note that in the first embodiment, information about the feature point distribution center may be transmitted to the other device, and the other device may be requested to update the shooting range and reshoot.

[0248] Changing the imaging range may change the distribution of matched external feature points. In this case, the new center of the distribution of external feature points may deviate from the center of the imaging range. In this case, the imaging range must be repeatedly updated.

[0249] The termination condition for updating the shooting range may be a certain number of times, with the termination being when the number of matching external feature points no longer increases. After updating the shooting range is completed, a determination process is performed based on the number of successful matches for the video pair with the largest number of matching external feature points.

[0250] (Variation 2) In this embodiment, after the trajectory approaching portion is identified and the coordinate transformation parameters are calculated, the area captured by the other terminal can also be determined. Therefore, when new coordinate transformation parameters are calculated using images from different locations and averaged to improve the accuracy of the coordinate transformation parameters, it is also possible to show the user the location where the other terminal captured images to guide them in taking additional images.

[0251] FIG. 34 is a diagram for explaining the processing of the second modified example of the sixth embodiment, showing an example in which a shooting target is displayed superimposed with a virtual object.

[0252] As shown in the example of FIG. 34, the video captured by the partner terminal at that location may also be displayed.

[0253] (Variation 3) In the above description, the same location is photographed in the trajectory approaching portion, but when photographing the same external area at a distance from the photographing location, the trajectories do not need to be close to each other.

[0254] For example, if the meeting place is a specific floor of a building, before entering the building, video of the building exterior or a familiar location near the building may be captured, and from those videos, video pairs with a certain number of matching external world feature points may be extracted and used to calculate the coordinate transformation parameters. In this case, the external world image captured by the first information terminal is designated as the third external world image, the external world feature points in the third external world image are designated as the third external world feature points, the external world image captured by the second information terminal is designated as the fourth external world image, and the external world feature points in the fourth external world image are designated as the fourth external world feature points. The calculation procedure for the coordinate transformation parameters can be performed in the same manner as above, by replacing the first external world image with the third external world image, the first external world feature points with the third external world feature points, the second external world image with the fourth external world image, and the second external world feature points with the fourth external world feature points. If the location is one where a general-purpose positioning system such as GPS is available, data on the location where the image was captured using the general-purpose positioning system can be sent from the device that first captured the location to the other device, and when the other device captures the image, a guide display of the captured image can be displayed using a virtual object superimposed on it, as shown in Figure 34. In this case, errors in the coordinate transformation parameters may be large due to the distance of the subject, and errors in the autonomous individual positioning using acceleration sensors 115A, 115B and gyro sensors 116A, 116B may increase as the distance traveled from the coordinate transformation parameter determination point increases. However, since the approximate capture position of the other device can be determined when additional images are captured near the meeting place, it is possible to easily capture the same location. Furthermore, even in locations where a general-purpose positioning system is available, it is worthwhile to calculate the coordinate transformation parameters because the effect of errors in the general-purpose positioning system can be reduced.

[0255] (Variation 4) In the above embodiment, the coordinate transformation parameters are determined using a pair of trajectory approaching part images captured at the trajectory approaching part. However, there may be an embodiment in which the first information terminal and the second information terminal each detect the terminal position and capture an external world image, but do not detect the distance to the external world feature point. In this case, index information linking the terminal position at the time of capture with the external world image is also stored in association with the external world image.

[0256] In this modification, a trajectory approaching portion video pair is determined (S01A to S04A and S01B to S04B in FIG. 24, however, distance measurement to external feature points is not performed in S01A and S01B), and index information of frames of the first information terminal included in the trajectory approaching portion video pair is output to the user of the first information terminal, and index information of frames of the second information terminal included in the trajectory approaching portion video pair is output to the user of the second information terminal as meeting support information (S10A, S10B). The frame index information may include, for example, the time the frame was captured. Each terminal can determine the location where the frame was captured from the frame index information.

[0257] (Effects of this Modification) According to this modification, the user of the first information terminal and the user of the second information terminal can easily find each other by returning to the location where the trajectory approach portion image pair was captured, i.e., the trajectory approach portion. In addition, since coordinate conversion is not performed, the processing load on each information terminal can be reduced.

[0258] Seventh Embodiment The seventh embodiment is an embodiment in which, in an area where an intra-area positioning system using external images is prepared, positioning is performed by the intra-area positioning system using the VPS (Visual Positioning System) technology used in the first embodiment based on landmark images of the starting point and passing points (hereinafter sometimes referred to as landmark points) while a user is moving from a starting point to a destination point. Positioning by the intra-area positioning system corrects accumulated errors in the autonomous positioning of the terminal.

[0259] The VPS technology of the present invention differs from conventional VPS technology in that it requires less information to be prepared in advance. Conventional VPS technology requires preparing a vast amount of library information so that the information terminal can determine its own terminal's location regardless of the external image captured by the information terminal. In contrast, the processing of the first to sixth embodiments of the present invention involves VPS positioning of the remote terminal's location based on the terminal's own terminal's world coordinate system, and uses images of external objects captured at the same location by both terminals, so no vast amount of library information is required. Furthermore, in this embodiment, only information about a small number of landmark locations placed in the external world is prepared, and the information terminal captures images of those landmark locations to determine its own terminal's location in the in-area world coordinate system, so no vast amount of library information is required.

[0260] Here, to clarify the concept, we will explain the "positioning system." First, in this invention, the "positioning system" refers to a system consisting of a method for measuring the position and orientation of an object and a coordinate system fixed to the external world that serves as a reference for numerically expressing the measured position and orientation. In this invention, the object whose position and orientation is to be measured is an information terminal.

[0261] A coordinate system fixed to the external world is called a "world coordinate system." A "world coordinate system" commonly used by all terminals is called a "universal world coordinate system." An example of a "universal world coordinate system" is a {latitude, longitude, altitude} coordinate system corresponding to positioning using satellite radio waves such as GPS. In the present invention, this "universal world coordinate system" and a positioning method using satellite radio waves such as GPS are collectively called a "universal positioning system." Strictly speaking, positioning using satellite radio waves such as GPS alone can determine the location of the terminal, but not its orientation. The orientation of the terminal can be determined by auxiliary measurements of geomagnetic fields or by estimating the orientation of the terminal from the movement and positional changes of the terminal. Well-known techniques can be used for the "universal positioning system" using satellite radio waves such as GPS, including this auxiliary positioning, and detailed explanations thereof will be omitted in the present invention.

[0262] In addition to the "general-purpose positioning system," this invention assumes that each terminal has its own terminal-specific positioning system. This terminal-specific positioning system is called the "terminal positioning system." The "world coordinate system" in the "terminal positioning system" is the "terminal world coordinate system." An example of a "terminal world coordinate system" is a "world coordinate system" that fixes the "terminal local coordinate system" at the moment the terminal is turned on relative to the external world. Here, the "terminal local coordinate system" is defined based on the terminal housing. For example, in the case of an HMD, the origin is the center of the terminal housing, and when worn by the user, the X axis is the direction in front of the user, the Y axis is the direction of the user's left hand, and the Z axis is the vertically upward direction. From the time the "terminal world coordinate system" is set, the position and orientation of the "terminal local coordinate system" within the "terminal world coordinate system," i.e., changes in the position and orientation of the terminal, are continuously measured. This measurement is called "autonomous positioning." As for an example of this "autonomous positioning" measurement, a detailed description has been given in the sixth embodiment for the case where the acceleration sensors 115A and 115B and the gyro sensors 116A and 116B are used.

[0263] In the seventh embodiment, the concepts of an "in-area positioning system" and a corresponding "in-area world coordinate system" are introduced. Positioning within an area uses a landmark image, which is an image of the outside world, and the terminal world coordinate system of the terminal capturing this landmark image is the "in-area world coordinate system." The terminal capturing the landmark image is called the "in-area reference terminal." The "in-area reference terminal" may be any terminal that measures its own position and orientation using the "in-area world coordinate system" as a reference, for example, by the "autonomous positioning" exemplified in the sixth embodiment, and may be composed of multiple terminals. However, measurement errors in the own terminal position of the "in-area reference terminal" capturing the landmark image result in errors in the accompanying information of the landmark image, which in turn results in errors in the "in-area positioning" using the landmark image. Therefore, it is desirable to use an "in-area reference terminal" with as high performance as possible.

[0264] Here, we will explain how the terminal world coordinate system is shared between terminals. Up until now, we have based the example of the terminal world coordinate system being a fixed terminal local coordinate system relative to the world at power-on, but it may also be aligned with the terminal world coordinate system of another terminal. To do this, an external world image is used to determine a coordinate system transformation parameter between terminals, using the method described in "Method (1) or (2) of acquiring the positional relationship between terminals" in the first embodiment. This coordinate system transformation parameter can be used to convert the terminal world coordinate system of the own terminal into the terminal world coordinate system of the other terminal, thereby switching the terminal world coordinate system. After switching the terminal world coordinate system, changes in the position and orientation of the own terminal relative to the new terminal world coordinate system are continuously measured using the method described in the sixth embodiment, etc.

[0265] Now, we will explain "in-area positioning" in an area where a landmark image is prepared. A terminal equipped with a "terminal positioning system" constantly measures its own position and orientation in its own "terminal world coordinate system" through "autonomous positioning." Therefore, for such a terminal, ideally, by using a landmark image from one location once to calculate the coordinate transformation parameters between the "in-area world coordinate system" and the "terminal world coordinate system" using the method described in "Method (1) or (2) for acquiring the positional relationship between terminals," the terminal's own position and orientation can always be recognized based on the "in-area world coordinate system." However, "autonomous positioning" typically accumulates errors over long periods of measurement. Therefore, if the distance traveled within the area is long, the coordinate transformation parameters between the "in-area world coordinate system" and the "terminal world coordinate system" are periodically recalculated using the landmark image, and the terminal's own position and orientation within the "in-area world coordinate system" based on the new landmark image is determined. This is "in-area positioning." This "intra-area positioning" corrects the cumulative measurement error of the changes in the terminal's position and orientation within the "intra-area world coordinate system" obtained by "autonomous positioning." Between "intra-area positioning" and the next "intra-area positioning," the terminal's position and orientation within the "intra-area world coordinate system" obtained by "autonomous positioning" are used for each process.

[0266] Here, in terms of the relationship with the hardware configuration, of the position and orientation sensors 135A and 135B, PP cameras 136A and 136B and PP ranging sensors 137A and 137B are used in processes for obtaining coordinate transformation parameters, such as obtaining the positional relationship between terminals and "intra-area positioning," while acceleration sensors 115A and 115B and gyro sensors 116A and 116B are used in "autonomous positioning."

[0267] Now, in the seventh embodiment, a virtual terminal called a "reference terminal" may be used, but the present invention is a method in which a user terminal obtains coordinate transformation parameters between "terminal world coordinate systems" from an external image and its accompanying information by the method described in "Method (1) or (2) for acquiring positional relationship between terminals" in the first embodiment, and recognizes the position and orientation of the own terminal in another "terminal world coordinate system," or vice versa, and guides the user to a destination position. The coordinate transformation parameters are the rotation Q ＡＢ and the origin difference O ＡＢ and the coordinate transformation formula was formula (17) (the inverse transformation was formula (18)).

[0268] Next, we will comment on the accompanying information for the external world image used to find the coordinate transformation parameters between device world coordinate systems. There are two types of accompanying information. Accompanying information type 1: Coordinate values ​​of external world feature points in the external world image in the device world coordinate system of the shooting device. Accompanying information type 2: Coordinates of the device position at the time of shooting in the device world coordinate system of the shooting device, and directional vector values ​​of the external world feature points as seen from the device position at the time of shooting.

[0269] In "Method for obtaining positional relationships between terminals (1)," a method for comparing external world images having accompanying information of type 1 was explained, and in "Method for obtaining positional relationships between terminals (2)," a method for comparing external world images having accompanying information of type 1 with external world images having accompanying information of type 2 was explained.

[0270] In this embodiment, a landmark image with accompanying information prepared at a landmark point within an area is compared with an external image captured by an information terminal at the landmark point. As in the other embodiments, the combination of the types of accompanying information of the landmark image and the external image captured by the information terminal is either a combination of Type 1 and Type 1, or a combination of Type 1 and Type 2. In the case of a combination of Type 1 and Type 2, the type of accompanying information of the landmark image can be either type.

[0271] In the explanation of "Method (1) or (2) for acquiring the positional relationship between terminals," the terminal world coordinate systems are distinguished by the subscripts "A" and "B," but when applied to this embodiment, either one is interpreted as the terminal world coordinate system of the "intra-area reference terminal" that is the basis of the intra-area world coordinate system. Also, the concept of the terminal position that captured the external world image appears, but in the case of the landmark image of this embodiment, it is interpreted as the position of the "intra-area reference terminal" that captured the landmark image.

[0272] The seventh embodiment differs from the first to sixth embodiments in that it addresses the cumulative error of "autonomous positioning." The first to sixth embodiments are applicable techniques that are applicable within a range in which the cumulative error of "autonomous positioning" does not become large and the distance from the shooting point of the external world image for which the coordinate transformation parameters are calculated is not large.

[0273] FIG. 35 is a diagram illustrating a use case of the guidance system according to the seventh embodiment.

[0274] Figure 35 shows the trajectory of a user of HMD 1A entering building A from outside at point A, which is the entrance to building A, passing through point B within building A, and moving toward point C, which is the target point within building A.

[0275] Because it is possible to receive positioning radio waves such as GPS outdoors, when searching for terminal location information on an information terminal, such as an HMD or smartphone, a general-purpose positioning system based on a general-purpose coordinate system such as GPS can be used. However, inside buildings or underground malls, the reception of positioning radio waves is poor, making it impossible to detect the position. Therefore, inside building A, the system switches to an in-area positioning system dedicated to that area, using an in-area world coordinate system and landmark images.

[0276] 36 is a diagram showing an example of a landmark image. A landmark image 300 shown in FIG. 36 is prepared at landmark point A in building A. When an image of the external environment overlapping with landmark image 300 is captured with an information terminal, the information terminal detects external environment feature points 301, 302, 303, and 304 included in landmark image 300.

[0277] In order to determine the relationship between the in-area world coordinate system and the terminal's own world coordinate system, the landmark image 300 is accompanied by information relating to external feature points 301, 302, 303, and 304 of the external subject that is captured in the landmark image 300 in advance.

[0278] FIG. 37 is a diagram showing an example of landmark image accompanying information. This corresponds to FIG. 7A of the first embodiment. The information items in this example are as follows: Landmark point position: Indicates the coordinate value of the landmark point. This landmark point position is also the position where the landmark image was captured. Shooting direction: Indicates the value of the directional vector from the landmark point position to the apparent center of the landmark image. When capturing images with a terminal, the shooting direction may be corrected by referring to the user's height and the deviation of the actual standing position from the target point position. Next landmark point position: Indicates the coordinate value of the next landmark point. Shooting direction at next landmark point: Indicates the value of the directional vector in which the next landmark image was captured from the next landmark point position. Information on multiple next landmark points, rather than just one, may be included. Information on all landmark points on the route may also be included. Coordinate value or directional vector value of external world feature point 1: In the case of type 1 accompanying information, indicates the coordinate value of external world feature point 1. In the case of type 2 accompanying information, indicates the directional vector value of external world feature point 1 as seen from the shooting point position, i.e., the landmark point position. Position of external feature point 1 in the image: Indicates the position of external feature point 1 in the landmark image. The unit may be pixels, or the vertical and horizontal widths of the image may be expressed as 1. Feature value of external feature point 1: Indicates the feature value of external feature point 1 used for feature point matching. The feature value is a vector value. Since the feature value changes depending on the image matching method, multiple types of feature values ​​may be included depending on the feature point matching method. If the feature value described in the accompanying information is not compatible with the feature point matching performed on the terminal, or if you want to get as close as possible to the processing in the captured data, the feature value of the external feature point is calculated from the image data of the landmark image. Note that the coordinate values ​​and directional vector values ​​in the accompanying information are based on the world coordinate system within the area.

[0279] For the sake of convenience, only two external feature points are described in the landmark image accompanying information in Fig. 37, but the number of external feature points required for one landmark image is at least three. It is preferable to describe data of four or more external feature points, since there is no guarantee that matching will be achieved depending on the shooting conditions at the information terminal.

[0280] If the accompanying information of the landmark video is Type 2, the accompanying information that the information terminal should prepare is Type 1. To obtain Type 1 accompanying information, it is necessary to measure the distance from the terminal position to the external feature point in addition to the processing for obtaining Type 2 accompanying information. If the party preparing the landmark video is, for example, a store, the user of the information terminal will be a customer, but from the perspective of minimizing the burden on the customer or not requiring high functionality from the customer's information terminal, a possible use case is that the store may prepare a landmark video with Type 1 accompanying information, and the customer's information terminal may only need to obtain Type 2 accompanying information.

[0281] In such a use case, the landmark image and the information accompanying the landmark image are acquired as area guidance information from an information provider, for example, the manager of building A, the operator of the shopping mall, or the store manager. The route guidance information may be received by the HMD 1A via the server 2. Furthermore, the information accompanying the landmark image may be configured as metadata of the landmark image information, as one file of information.

[0282] In addition to the use case of landmark video as area guidance information as described above, there is also a use case of landmark video as tracking information. Tracking information corresponds to area guidance information. For example, if a friend is late to their destination and loses track of their route, they can record some footage of their route as landmark video with accompanying information on their own device and provide it to their friend. Another use case is to record video of their own journey as video with accompanying information on their own device in case they lose track of their route home, and then refer to it later. In such cases, since it is not certain whether the accompanying information will be used, it is possible to consider Type 2 accompanying information to minimize the effort required to obtain it.

[0283] After generating conversion information between the in-area world coordinate system and the terminal world coordinate system at a landmark point (e.g., point A in FIG. 35 ), the information terminal reads the next landmark point (e.g., point B in FIG. 35 ) from the landmark image accompanying information and guides the user to the next landmark point (e.g., point B in FIG. 35 ). The information terminal continues autonomous positioning. Then, at the next landmark point (e.g., point B in FIG. 35 ), the information terminal shows the shooting point and shooting direction to the user, identifies external feature points, and updates the conversion information between the in-area world coordinate system and the terminal world coordinate system, thereby correcting the position and attitude measurement by autonomous positioning. This process is repeated up to the target point (e.g., point C in FIG. 35 ). As examples of guidance displays, the examples of guidance displays shown in FIGS. 30 and 31 described in the sixth embodiment may be used. Replacing the “estimated partner terminal position” in FIGS. 30 and 31 with “next landmark point” can be used as an example of guidance displays in this embodiment.

[0284] Here, a specific example of position and attitude correction by autonomous positioning will be described. First, at a certain landmark point N, coordinate transformation parameters {Q （Ｎ） ＡＢ , O （Ｎ） ＡＢ Here, the subscript A represents the in-area world coordinate system, and the subscript B represents the terminal world coordinate system. Next, move to the next landmark point N+1, and similarly use the external world image to obtain the coordinate transformation parameters {Q （Ｎ＋１） ＡＢ , O （Ｎ＋１） ＡＢ} is calculated. Ideally, this coordinate transformation parameter should be the same regardless of which landmark point it is calculated at, but in reality, errors in the autonomous positioning of the information terminal cause errors in the accompanying information of the external world image measured at landmark point N+1, resulting in errors in the coordinate transformation parameters. Note that although there are errors in the autonomous positioning of the reference terminal, position information within the area is determined based on the measurement of the reference terminal, and so errors in the autonomous positioning of the reference terminal are also corrected as errors in the autonomous positioning of the information terminal.

[0285] The error in the accompanying information of the external world image on the information terminal side due to an error in autonomous positioning means that when the terminal world coordinate system at landmark point N is used as a reference, the terminal world coordinate system at landmark point N+1 appears to be shifted by the amount of the error in autonomous positioning. Now, let the rotation of the terminal local coordinate system relative to the terminal world coordinate system of the information terminal measured from the reference time be Qm(N), Qm(N+1), and the origin movement amount be Om(N), Om(N+1). The N and N+1 in parentheses represent landmark points. In contrast, if there is no error in autonomous positioning, the rotation of the terminal local coordinate system from landmark point N to N+1 is Qt(N, N+1), and the origin movement amount is Ot(N, N+1). In this case, the error in autonomous positioning is expressed as the coordinate transformation parameter {Q （Ｎ）（Ｎ＋１） , O （Ｎ）（Ｎ＋１）}, the following equation is obtained: Q （Ｎ）（Ｎ＋１） = Qm(N+1) ＊ Qt(N, N+1)Qm(N)...(45) O （Ｎ）（Ｎ＋１） =Ot(N,N+1)+Om(N)-Om(N+1)...(46)

[0286] The relationship between these parameters and the coordinate transformation parameters between the in-area world coordinate system and the terminal coordinate system at landmark point N+1, which are obtained while including the influence of errors in autonomous positioning, is expressed by the following equation: Q （Ｎ＋１） ＡＢ =Q （Ｎ）（Ｎ＋１） Q （Ｎ） ＡＢ ... (47) O （Ｎ＋１） ＡＢ =O （Ｎ） ＡＢ +O （Ｎ）（Ｎ＋１） ...(48)

[0287] From these relational expressions, the ideal value when there is no error in autonomous positioning can be calculated as follows: Qt(N, N+1)=Qm(N+1)Q （Ｎ＋１） ＡＢ Q （Ｎ） ＡＢ ＊ Qm(N) ＊ ...(49) Ot(N, N+1)=Om(N+1)-Om(N)+O （Ｎ＋１） ＡＢ -O （Ｎ） ＡＢ...(50)

[0288] Therefore, the correction at the landmark point N+1 is performed as follows: (corrected Qm(N+1)) = (uncorrected Qm(N+1)) Q （Ｎ＋１） ＡＢ Q （Ｎ） ＡＢ ＊ ...(51) (Om(N+1) after correction) = (Om(N+1) before correction) + O （Ｎ＋１） ＡＢ -O （Ｎ） ＡＢ ...(52)

[0289] At the same time as this correction, the coordinate transformation parameters at landmark point N+1 are rewritten to those of landmark point N. Ultimately, at the same time as correcting the autonomous positioning of the information terminal, the coordinate transformation parameters are rewritten to the values ​​of the coordinate transformation parameters calculated at the first landmark point after entering the area. Note that the position of the information terminal when calculating the coordinate transformation parameters at the landmark point does not need to be strictly matched to the coordinate values ​​of the reference terminal that captured the landmark image, as long as it is located at a position where the same external subject as the landmark image can be captured.

[0290] The points where the equations (51) and (52) are corrected are determined by the information terminal using the coordinate transformation parameters {Q （Ｎ＋１） ＡＢ , O （Ｎ＋１） ＡＢ} is ideally the point where the outside world image for determining the position {{position}} is captured. However, if the information terminal user is in a hurry to move, the information terminal captures the outside world image and calculates the coordinate transformation parameters while moving, and when the calculation of the coordinate transformation parameters is completed, performs autonomous positioning correction (corresponding to equations (51) and (52)) for the value of that point. Naturally, in this case, the correction is performed at a point that is deviated from the ideal point, so an error due to the correction occurs, but as long as the error is not too large, there is no problem in searching for the next landmark point.

[0291] FIG. 38 is a flowchart showing the flow of processing of the guidance system in the seventh embodiment.

[0292] Assume that an information terminal, for example, HMD 1A, arrives at entrance A of building A while detecting its position using a general-purpose positioning system. From here on, inside building A, the terminal position detection and guidance service is switched to an in-area positioning system using landmark images based on an in-area world coordinate system. Therefore, the user of the information terminal performs an operation to switch the information terminal from the general-purpose positioning system used by the navigation system to the in-area positioning system. For example, the positioning system may be switched by starting an interior guidance app on the information terminal that uses the in-area positioning system of building A. It is assumed that autonomous positioning by the information terminal continues from the area of ​​the general-purpose positioning system.

[0293] First, the information terminal requests area guide information for building A from server 2 and receives the area guide information from server 2 (S201). In the area guide information, each point in building A is associated with a coordinate value in the area world coordinate system.

[0294] The server 2 receives a request for area guidance information from the information terminal and transmits the area guidance information to the information terminal (S202).

[0295] When the information terminal receives the area guidance information, it captures an image of the external subject of the first landmark image and measures the distance to the external feature point. Here, it is assumed that the accompanying information to be measured by the information terminal is Type 1, but if the accompanying information to be measured by the information terminal is Type 2, it is sufficient to measure only the direction vector without measuring the distance to the external feature point (S203). Note that it is acceptable for the accompanying information of the landmark image to be a mixture of Type 1 and Type 2.

[0296] Here, a method for recognizing an external subject in the first landmark image (in the case of Figure 35, the landmark image of point A) will be described. The information terminal does not know the location described in the area world coordinate system until it obtains coordinate transformation parameters with the area world coordinate system at the first landmark point. Therefore, for landmark points near the entrance of a building that may be the first landmark point, coordinate values ​​in the general-purpose world coordinate system may also be described in the area guidance information. Even if reception of radio waves such as GPS radio waves is lost, the information terminal can determine its own position and orientation in the general-purpose world coordinate system by autonomous positioning as long as the distance is not too long. Alternatively, the first landmark image may be displayed on the terminal, and the coordinate transformation parameters at the first landmark point may be set by receiving visual guidance from the terminal user.

[0297] The user uses the outer camera 111A of the information terminal, for example, the HMD 1A, to capture an image of an external world subject captured in the landmark image 300 prepared at entrance A, and measures the distance from the terminal position to an external world feature point consisting of a part of the external world subject (S203). This process is the same as step S01A in the first embodiment. Distance measurement may be performed using distance measurement sensors 113A and 113B such as LiDAR or TOF, or a moving stereo method using multiple images captured at different positions. Note that distance measurement is not performed if the accompanying information of the external world image to be prepared by the information terminal is Type 2. Alternatively, if distance measurement of the external world feature point is performed after feature point matching, distance measurement at this stage may not be performed.

[0298] Next, the information terminal performs a matching process for external feature points (S204). The landmark image data and the associated information data may be included in the area guidance information received in step S201 as data on the first landmark location candidate, or may be included in the route information received in step S209 after the route is set. Although not shown, if only landmark location information has been received, the landmark image data and the associated information data to be used for video comparison may be received from a server at this stage. Hereinafter, information transmission and sharing are assumed to be performed via the network communication devices 120A and 220, and an explicit description thereof will be omitted. The landmark image associated information may be generated as a two-dimensional code, and the information terminal may be configured to read the landmark image associated information when capturing the landmark image 300. The process of step S204 is the same as step S03A in the first embodiment. More specifically, the feature amounts of the external world feature points captured in the external world image of the external world subject captured by the information terminal are compared with the feature amounts of the external world feature points recorded in the landmark image accompanying information, and matching of the external world feature points is performed. Hereinafter, in descriptions in which the information terminal and the server are the processing entities, the processing entities at the hardware component level shown in Figures 2 and 3 are the processors 125A and 225.

[0299] Here, the information terminal determines whether the number of matching external field feature points is equal to or greater than a predetermined number (S205). This process is the same as step S04A in the first embodiment.

[0300] If the determination in step S205 is negative (S205: No), the match with the landmark video image is insufficient. In other words, when the external subject captured in the landmark video was photographed with the information terminal, even if the same external subject was photographed, the photographing position and photographing direction were significantly different from when the landmark video was photographed (to the extent that matching of external feature points cannot be performed). Therefore, the process returns to step S203, and the external subject captured in the landmark video is photographed again to generate a new external video.

[0301] If the determination in step S205 is positive (S205: Yes), the information terminal measures the distance between the information terminal and the external feature point on the matched external subject (S206). This process is the same as step S05A in the first embodiment. If distance measurement has been performed in step S203, distance measurement at this stage may be omitted. Furthermore, if the accompanying information of the external scene image that the information terminal should prepare is Type 2, distance measurement is not performed.

[0302] In the distance measurement in step S206, if the information terminal has not moved from the shooting point of the video used for matching, distance measurement can be performed using the ranging sensor 113A such as LiDAR or TOF. If the information terminal moves and distance measurement to the external world feature point becomes impossible using the ranging sensor 113A, the information terminal captures multiple videos in advance using the outer camera 111A by shifting the shooting point within a range where the same external world area can be captured, and measures the distance from the shooting point to the external world feature point using the moving stereo method.

[0303] Next, the information terminal uses the external image and the measured associated information, and the landmark image and associated information at the landmark point to determine coordinate transformation parameters between the area coordinate system and the terminal coordinate system using the "method (1) or (2) for acquiring the positional relationship between terminals" according to the type of associated information.The information terminal uses these coordinate transformation parameters to acquire the terminal position and orientation in the area coordinate system (S207).If this is the first landmark point for the information terminal since entering the area, the information terminal sets this terminal position and orientation as initial values; if not, it corrects the terminal position and orientation.Thereafter, the position and orientation within the area are measured using autonomous positioning.

[0304] Here, the information terminal determines whether or not a destination point within the area has already been set (S208).

[0305] If step S208 is positive, the process proceeds to step S211. If step S208 is positive, the vehicle is at the second or subsequent landmark, or is at the first landmark, but the initial area guidance information received in step S201 includes route information, and route selection has been completed. There may be cases where the information request in step S201 requests a route to a specific store.

[0306] If step S208 is negative, the process proceeds to step S209.

[0307] In step S209, the information terminal sets a destination point by referring to the area guidance information, transmits the destination point information and its own terminal position information to the server 2, requests route information to the destination point (S209), and receives the information transmitted from the server 2 in response to the request. Then, the process proceeds to step S211. If the area guidance information includes map information of the area, the area map may be displayed on the information terminal to support the user in setting a destination point.

[0308] The server 2 receives the current terminal position and destination point information from the information terminal, calculates a route from the current location to the destination point, and transmits the route to the information terminal together with landmark points, landmark images, and accompanying information (S210). All landmark points, landmark images, and accompanying information within the area may be transmitted to the information terminal in step S201, and in this step, only route information indicating which landmark points to follow may be transmitted.

[0309] In step S211, the information terminal recognizes the next landmark point, calculates the direction of the next landmark point from the current terminal position, generates guidance information to the next landmark point, and outputs the guidance information (S211). Examples of the guidance information output here correspond to the screen display examples in Figures 30 and 31.

[0310] There are several variations in the method for recognizing the next landmark point. When landmark points, landmark images, and accompanying information from the starting point to the destination are received together with route setting after the destination within the area has been decided, for example, the next landmark point information written in the accompanying information is read. When the destination is decided at the time of entering the area, the next landmark point information may be written in the accompanying information of the first landmark image. Furthermore, landmark point information to be followed in order may be received in list form together with the route setting. Alternatively, nearby landmark points may be written in the accompanying information of the landmark image, and the information terminal may select one on the set route as the next landmark point.

[0311] After step S211, the information terminal proceeds to step S212. In step S212, the information terminal moves toward the next landmark point and continues to measure the position and orientation of the terminal itself by autonomous positioning (S212).

[0312] In step S213, the information terminal determines whether or not the destination point has been reached (S213).

[0313] If the result of step S213 is affirmative (S213: Yes), the information terminal ends the process.

[0314] If the result of step S213 is negative (S213: No), the information terminal proceeds to step S214 and determines whether the next landmark point has been reached (S214).

[0315] If the result of step S214 is negative (S214: No), the information terminal returns to step S211 and continues the process. At this time, the output of guidance information is also updated in accordance with the change in the terminal position.

[0316] If step S214 is positive (S214: Yes), the information terminal returns to step S203 and starts processing at the new landmark point.

[0317] (Effects of this embodiment) In this embodiment, an external subject captured in a landmark image prepared at a landmark point within the area to be used is captured by an information terminal, and the position of the information terminal in the world coordinate system within the area is estimated by comparing an accompanying image of the landmark image related to an external feature point in the image with accompanying information of the external image captured by the information terminal. Position estimation during movement between landmark points utilizes the autonomous positioning of the information terminal. This makes it possible to easily estimate the position of the information terminal even in areas where general-purpose positioning systems such as GPS cannot be used, without having to prepare a huge amount of library images for position estimation in advance.

[0318] (Variation 1) In the above embodiment, the starting point and the destination point are fixed, but if you want to go to various places, such as an event venue, you may use a positioning method that uses images of landmarks placed within the area to correct your position and posture within that area. Figure 39 is a diagram explaining Variation 1 of the seventh embodiment.

[0319] For example, when landmark points are set as shown in Figure 39, the other landmark image positions described in the landmark image accompanying information may be all landmark images. Alternatively, information on landmark image positions may be obtained as list information from the area manager in addition to the landmark image data. Information on landmark image positions may also be included in the area guidance information.

[0320] When the terminal user moves within the area and passes near a landmark, the terminal user can refer to the area guidance information and correct the position and attitude measurements using autonomous positioning at the location where the landmark image was taken, allowing the terminal user to go to the location they want to go to within the area while maintaining the positioning accuracy of autonomous positioning.

[0321] (Variation 2) In the basic form of this embodiment, the first landmark point is set within the area, but the first landmark point may also be set at a location where positioning by a general-purpose positioning system such as GPS is available. To guide the user to this landmark point, guidance information is included in map information corresponding to the general-purpose positioning system. This allows for a smooth transition from the general-purpose positioning system to the in-area positioning system, improving user convenience. Figures 40 and 41 are diagrams explaining Variation 2 of the seventh embodiment.

[0322] Specifically, as shown in Fig. 40, in general-purpose positioning system-based map information, the shooting position and shooting direction of a landmark image that serves as an entry point of the intra-area positioning system of the use area are provided as general-purpose positioning system-based information. An example of the landmark image is an image of the entrance to a building in the use area, as shown in Fig. 41. At this entrance, the information terminal communicates with the use area management server 2 and receives the intra-area guide information at step S201. Then, with this building entrance as the first landmark point, the information terminal shoots an external image of the part that is the subject of the landmark image and determines the transformation relationship between the coordinate system of the intra-area world coordinate system and the terminal world coordinate system using the method described above.

[0323] After entering the area of ​​the intra-area world coordinate system, the user moves to the destination using the landmark images placed within the area according to the processing procedure already described.

[0324] (Variation 3) It is preferable that external feature points in the landmark image remain unchanged as much as possible. For example, movable signboards and electronic signboards change position and appearance depending on the time of day, making them difficult to use as external feature points.

[0325] Therefore, external world feature points may be selected visually, or multiple images of a location that is a candidate subject for the landmark image may be taken at different times, external world feature point matching may be performed between those images, and external world feature points for which matching is established may be included as accompanying information. This improves the success rate of external world feature point matching.

[0326] (Variation 4) In areas of building A where the density of landmark images is low, the server 2 may add landmark images posted from an information terminal. This addition mode includes adding a shooting direction for the same external subject. Furthermore, if the travel distance between landmark points is long, video data along the route may be posted to the server 2 from another information terminal, or the route information may be posted. The route information may be used to set a point where posts are solicited. This allows for the enhancement of area guidance information while using the guidance system.

[0327] (Variation 5) In the basic form of this embodiment, it is assumed that there are no errors in the landmark image information, and that all errors in position measurement are due to the autonomous positioning of the information terminal. However, as in Variation 4, depending on the landmark images posted by users and the performance of the reference terminal used by the area manager, it may be better to also correct errors on the landmark image side. For this reason, information on the landmark image side is corrected based on multiple error data from the information terminal for the landmark image under consideration.

[0328] The error data referred to here is the mismatch of the coordinate transformation parameters that should be matched, calculated by the information terminal. Specifically, when one information terminal is focused on and the target point for which error correction is considered is N+1, the coordinate transformation parameters {Q （Ｎ）（Ｎ＋１） , O （Ｎ）（Ｎ＋１）}, and the errors are named ΔQ and ΔO, and the equations (47) and (48) are transformed to obtain the following equations: ΔQ=Q （Ｎ）（Ｎ＋１） =Q （Ｎ＋１） ＡＢ Q （Ｎ） ＡＢ ＊ ...(53) ΔO=O （Ｎ）（Ｎ＋１） =O （Ｎ＋１） ＡＢ -O （Ｎ） ＡＢ ...(54)

[0329] The average [<ΔQ>] and <ΔO> of this error measured by many information terminals are calculated. Here, [·] represents normalization with a norm of 1. Since it is expected that errors caused by individual information terminals will cancel out when averaged, this average value can be considered the error on the landmark image side. This average value can be interpreted as a coordinate transformation parameter to the true coordinate, so this average value is used to correct the accompanying information of the landmark image and the coordinate values ​​within the area around the landmark image. Specifically, the formula is as follows: (corrected coordinate value) = [<ΔQ>] ＊ (Coordinate value before correction) [<ΔQ>] - <ΔO> (55) (Direction vector value after correction) = [<ΔQ>] ＊ (Direction vector value before correction) [<ΔQ>] ... (56)

[0330] In the procedure of the flow chart, each information terminal calculates the coordinate transformation parameters in step S207 and simultaneously transmits the errors of the coordinate transformation parameters in equations (53) and (54) to the server 2. When the server 2 accumulates the error data and determines that the data on the landmark image side should be corrected, it corrects the area guidance information. This makes it possible to improve the accuracy of the position data in the area guidance information while using the guidance system.

[0331] (Example 6) Assuming that the data on the landmark image side also contains errors, the degree of correction of the autonomous positioning on the information terminal side may be changed depending on the degree of calibration on the landmark image side. For example, in the case of landmark images posted by users, the images are ranked according to the cumulative number of corrections, and the smaller the cumulative number of corrections, the lighter the degree of correction of the autonomous positioning on the information terminal side.

[0332] The degree of correction for the autonomous positioning is determined as follows, for example. The basic formulas for correction are formulas (51) and (52). Of these, the correction for formula (51) is （Ｎ＋１） ＡＢ Q （Ｎ） ＡＢ ＊ represents a rotation of a certain angle λ. If the degree of correction is controlled by the parameter η (0≦η≦1), then in equation (51), Q （Ｎ＋１） ＡＢ Q （Ｎ） ＡＢ ＊ In equation (52), the correction part O of the origin difference is （Ｎ＋１） ＡＢ -O （Ｎ） ＡＢ η(O （Ｎ＋１） ＡＢ -O （Ｎ） ＡＢ This allows for more realistic correction of dead-end positioning.

[0333] Although the embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain each embodiment, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of each embodiment will not be impaired even if different ones are used.

[0334] The programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each process is performed may also be changed.

[0335] Some or all of the functions of the above-described embodiments may be implemented in hardware, for example, by designing them as integrated circuits. Alternatively, they may be implemented in software by a microprocessor unit, CPU, or the like interpreting and executing an operating program that implements each function. Furthermore, the scope of software implementation is not limited, and hardware and software may be used together. Alternatively, some or all of the functions may be implemented in a server. The server may be any type of server, such as a local server, cloud server, edge server, or internet service, as long as it can execute functions in cooperation with other components via communications. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communications network.

[0336] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all of the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

[0337] The above embodiment includes the following aspects.

[0338] (Supplementary Note 1) An information terminal comprising: a processor; and a camera, wherein the camera generates an external world image in which an external world subject is reflected; the processor acquires feature amounts of each of the plurality of external world feature points reflected in a landmark image previously captured of the external world subject and coordinate values ​​of each of the plurality of external world feature points in an area world coordinate system; calculates values ​​in the terminal world coordinate system of the information terminal of direction vector values ​​from a terminal position of the information terminal toward each of the plurality of external world feature points reflected in the external world image; an information terminal that calculates coordinate transformation parameters between the terminal world coordinate system of the information terminal and the intra-area world coordinate system based on a result of comparing feature amounts of each of the plurality of external world feature points reflected in the external world image with feature amounts of each of the plurality of external world feature points reflected in the landmark image, a direction vector value from the terminal position of the information terminal toward each of the plurality of external world feature points reflected in the external world image, and coordinate values ​​in the intra-area world coordinate system of the plurality of external world feature points reflected in a landmark image obtained by previously capturing the external world subject; calculates a terminal position of the information terminal in the intra-area world coordinate system based on the coordinate transformation parameters; and outputs guidance information that guides the information terminal to a target point defined in the intra-area world coordinate system.

[0339] (Supplementary Note 2) An information terminal comprising: a processor; a camera; and a distance measurement sensor, wherein the camera generates an external world image in which an external world object is reflected; the distance measurement sensor measures individual distances from a position of the information terminal to a plurality of external world feature points that are part of the external world object; the processor acquires feature amounts of each of the plurality of external world feature points reflected in a landmark image previously captured of the external world object and coordinate values ​​of each of the plurality of external world feature points in an area world coordinate system, or coordinate values ​​of a reference terminal position in an area world coordinate system of a reference terminal that captured the landmark image, and direction vector values ​​from the reference terminal position to the plurality of external world feature points reflected in the landmark image; and calculates coordinate values ​​in the terminal world coordinate system of the information terminal for each of the plurality of external world feature points reflected in the external world image. an information terminal that calculates coordinate transformation parameters between the terminal world coordinate system of the information terminal and the intra-area world coordinate system based on a result of comparing each feature amount of the plurality of external world feature points reflected in the external world image with each feature amount of the plurality of external world feature points reflected in the landmark image, coordinate values ​​in the terminal world coordinate system of the information terminal of the plurality of external world feature points reflected in the external world image, coordinate values ​​in the intra-area world coordinate system of the plurality of external world feature points reflected in a landmark image previously captured of the external subject, or coordinate values ​​of a reference terminal position in the intra-area world coordinate system of a reference terminal that captured the landmark image, and a direction vector value from the reference terminal position to the plurality of external world feature points reflected in the landmark image; calculates a terminal position of the information terminal in the intra-area world coordinate system based on the coordinate transformation parameters; and outputs guidance information that guides the information terminal to a target point defined in the intra-area world coordinate system.

[0340] (Supplementary Note 3) A guidance method using an information terminal, comprising the steps of: generating an external world image in which an external world subject is reflected, and measuring, for a plurality of external world feature points that are part of the external world subject, a direction vector value in the terminal world coordinate system of the information terminal from the position of the information terminal to each of the external world feature points; acquiring a feature amount of each of the plurality of external world feature points reflected in a landmark image obtained by capturing the external world subject in advance, and a coordinate value in the in-area world coordinate system of each of the plurality of external world feature points; and calculating a coordinate transformation parameter between the terminal world coordinate system and the in-area world coordinate system based on a result of comparing the feature amount of each of the plurality of external world feature points reflected in the external world image with the feature amount of each of the plurality of external world feature points reflected in the landmark image, a direction vector value from the terminal position of the information terminal to each of the plurality of external world feature points, and the coordinate value in the in-area world coordinate system of each of the plurality of external world feature points reflected in the landmark image; a step of calculating a terminal position of the information terminal in the intra-area world coordinate system using the coordinate transformation parameters; and a step of outputting guidance information for guiding the information terminal to a target point defined in the intra-area world coordinate system.

[0341] 1: Vehicle 1A: In-vehicle terminal 1B: HMD 1C: Smartphone 2: Server 3: Network 4: Access point 100: Meeting support system 111A: Outer camera 111B: Outer camera 111C: Outer camera 112C: Inner camera 113A: Distance measurement sensor 113B: Distance measurement sensor 113C: Distance measurement sensor 115A: Acceleration sensor 115B: Acceleration sensor 115C: Acceleration sensor 116A: Gyro sensor 116B: Gyro sensor 116C: Gyro sensor 117A: Geomagnetic sensor 117B: Geomagnetic sensor 117C: Geomagnetic sensor 118A: Positioning sensor 118B: Positioning sensor 118C: Positioning sensor 119A: Display 119B: Display 119C: Display 120A: Network communication device 120B: Network communication device 120C: Network communication device 121A: Microphone 121B: Microphone 121C: Microphone 122A: Speaker 122B: Speaker 122C: Speaker 123A: Antenna 123B: Antenna 123C: Antenna 124: Touch panel 125A: Processor 125B: Processor 125C: Processor 126A: Program 126B: Program 126C: Program 127A: Data 127B: Data 127C: Data 128A: Memory 128B: Memory 128C: Memory 131C: Telephone network communication device 135A : Position and orientation sensor 135B : Position and orientation sensor 135C : Position and orientation sensor 136A : PP camera 136B : PP camera 136C : PP camera 137A : PP distance measurement sensor 137B : PP distance measurement sensor 137C : PP distance measurement sensor 140A : Bus 140B : Bus 140C : Bus 220 : Network communication device 223 : Antenna 225 : Processor 226 : Program 227 : Data 228 : Memory 240 : Bus300: Landmark image 301: External world feature point 302: External world feature point 303: External world feature point 304: External world feature point 1000: First external world image 1001: Second external world image 1010A: First external world feature point 1010B: First external world feature point 1010C: First external world feature point 1011A: Second external world feature point 1011B: Second external world feature point 1011C: Second external world feature point 1200: External world image 1210: Virtual object 1211: Virtual object 1300: External world image 1310: Virtual object 1311: Virtual object

Claims

1. An information terminal comprising: a processor; and a camera, wherein the camera generates an external world image in which an external world subject is reflected; the processor acquires feature amounts of each of the plurality of external world feature points reflected in a landmark image previously captured of the external world subject and coordinate values ​​of each of the plurality of external world feature points in an in-area world coordinate system; calculates values ​​in the terminal world coordinate system of the information terminal of directional vector values ​​from a terminal position of the information terminal toward each of the plurality of external world feature points reflected in the external world image; an information terminal which calculates coordinate transformation parameters between the terminal world coordinate system of the information terminal and the intra-area world coordinate system based on a result of comparing each of the feature amounts of the plurality of external field feature points reflected in the external field image with each of the feature amounts of the plurality of external field feature points reflected in the landmark image, values ​​of directional vector values ​​from a terminal position of the information terminal toward each of the plurality of external field feature points reflected in the external field image in a terminal world coordinate system of the information terminal, and coordinate values ​​in the intra-area world coordinate system of the plurality of external field feature points reflected in a landmark image obtained by previously capturing an external field subject, calculates a terminal position of the information terminal in the intra-area world coordinate system based on the coordinate transformation parameters, and outputs guidance information for guiding the information terminal to a target point defined in the intra-area world coordinate system.

2. An information terminal comprising a processor, a camera, and a distance measurement sensor, wherein the camera generates an external world image in which an external world subject is reflected, and the distance measurement sensor measures individual distances from the position of the information terminal to each of a plurality of external world feature points that are part of the external world subject, and the processor acquires feature amounts of each of the plurality of external world feature points reflected in a landmark image previously captured of the external world subject and coordinate values ​​of each of the plurality of external world feature points in an in-area world coordinate system, or coordinate values ​​of a reference terminal position in an in-area world coordinate system of a reference terminal that captured the landmark image and directional vector values ​​from the reference terminal position to the plurality of external world feature points reflected in the landmark image, and calculates coordinate values ​​in the terminal world coordinate system of the information terminal for each of the plurality of external world feature points reflected in the external world image, an information terminal which calculates coordinate transformation parameters between the terminal world coordinate system of the information terminal and the in-area world coordinate system based on a result of comparing each feature amount of the plurality of external field feature points reflected in the external field image with each feature amount of the plurality of external field feature points reflected in the landmark image, and based on coordinate values ​​in the terminal world coordinate system of the information terminal of the plurality of external field feature points reflected in the external field image, and coordinate values ​​in the in-area world coordinate system of the plurality of external field feature points reflected in a landmark image obtained by previously capturing an external field subject, or coordinate values ​​of a reference terminal position in the in-area world coordinate system of a reference terminal which captured the landmark image, and a direction vector value from the reference terminal position to the plurality of external field feature points reflected in the landmark image, calculates a terminal position of the information terminal in the in-area world coordinate system based on the coordinate transformation parameters, and outputs guidance information for guiding the information terminal to a target point defined in the in-area world coordinate system.

3. An information terminal as described in claim 1 or claim 2, further comprising a position and orientation sensor that detects an amount of movement of the information terminal, wherein the position and orientation sensor detects position and orientation change information of the terminal local coordinate system of the information terminal with respect to the terminal world coordinate system, and the processor updates the terminal position of the information terminal in the in-area world coordinate system calculated based on the external world image, based on position and orientation change information of the terminal local coordinate system of the information terminal with respect to the terminal world coordinate system detected in accordance with the amount of movement of the information terminal from the calculated terminal position.

4. An information terminal as claimed in claim 1 or 2, further comprising a communications interface for communicating with a server and a display, wherein the processor displays a map on the display based on map information defined in the in-area world coordinate system obtained from the server, accepts input for setting a destination point on the map, determines a relay point for capturing the next landmark image along a route to the destination point, starting from a point in the in-area world coordinate system where the external subject first captured in the landmark image is located, and outputs the determined relay point as guidance information for guiding the user to the relay point.

5. A guidance method using an information terminal, comprising the steps of: an information terminal generating an external world image in which an external object is reflected, and measuring a direction vector value in the terminal world coordinate system of the information terminal from the position of the information terminal to each of a plurality of external world feature points that are part of the external world object; acquiring a feature value of each of the plurality of external world feature points reflected in a landmark image previously captured of the external world object and a coordinate value in the in-area world coordinate system of each of the plurality of external world feature points; and calculating a coordinate transformation parameter between the terminal world coordinate system and the in-area world coordinate system based on a result of comparing the feature value of each of the plurality of external world feature points reflected in the external world image with the feature value of each of the plurality of external world feature points reflected in the landmark image, a direction vector value from the terminal position of the information terminal to each of the plurality of external world feature points, and a coordinate value in the in-area world coordinate system of each of the plurality of external world feature points reflected in the landmark image; A guidance method using an information terminal, comprising: a step of calculating a terminal position of the information terminal in the intra-area world coordinate system using the coordinate transformation parameters; and a step of outputting guidance information for guiding the information terminal to a target point defined in the intra-area world coordinate system.

Images