Information terminal, rendezvous support method, and rendezvous support system

Abstract

This information terminal: captures first outside-world video within a rendezvous site area while moving within the rendezvous site area; creates first outside-world feature point information indicating the relative positional relationship between a first outside-world feature point in the first outside-world video and the information terminal; receives a plurality of second outside-world videos captured by an other-party information terminal while moving within the rendezvous site area and a plurality of pieces of second outside-world feature point information indicating the positional relationships between second outside-world feature points in the plurality of second outside-world videos and the other-party information terminal at the time the video was captured; selects one second outside-world video from the plurality of second outside-world videos; and, by comparing the first outside-world video and the first outside-world feature point information with the selected second outside-world video and the corresponding second outside-world feature point information, calculates the inter-terminal positional relationship between the information terminal and the other-party information terminal to guide a user of the information terminal to the terminal position of the other-party information terminal.

Description

Information terminal, meeting support method, and meeting support system

[0001] The present invention relates to an information terminal, a meeting support method, and a meeting support system.

[0002] As a technology for supporting meetings, the information transmission / reception device in Patent Document 1 is described as comprising "a current location acquisition means provided on the side of a first user who is meeting a second user at a predetermined meeting place, and which includes: a current location acquisition means for acquiring the current location of the first user; a meeting place acquisition means for acquiring the meeting place for the first user and the second user; a traveling direction prediction means for predicting the traveling direction of the first user when arriving at the meeting place based on the positional relationship between the current location of the first user and the meeting place; an imaging command generation means for generating an imaging command for capturing an image of the meeting place in a predetermined imaging direction based on the traveling direction predicted by the traveling direction prediction means; and a transmission / reception means for sending the generated imaging command to the second user (abstract excerpt)."

[0003] Japanese Patent Application Laid-Open No. 2005-17074

[0004] In the information transmission and reception device of Patent Document 1, a user heading to a meeting place by car can tell that they have arrived near the meeting place by comparing the image taken by the other user at the meeting place with the scenery in front of the car, but there is a problem in that they cannot know the exact location of the other user.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an information terminal, a meeting support method, and a meeting support system that accurately notify a user of the location of a target location such as a meeting place.

[0006] In order to solve the above-mentioned problems, the present invention has the configuration described in the claims. For example, an information terminal includes a processor, a camera, and a communication interface, wherein the camera captures a first external image within a meeting place area while the information terminal is moving within the meeting place area, the processor generates first external feature point information indicating a relative positional relationship between a first external feature point in the first external image and the information terminal based on the first external image, and a plurality of second external images captured by another information terminal while moving within the meeting place area via the communication interface, and a plurality of second external feature point information indicating a relative positional relationship between the first external feature point in the first external image and the information terminal based on the plurality of second external images. The device receives a plurality of pieces of second external environment feature point information indicating the positional relationship between second external environment feature points in the external environment image and the other person's information terminal at the time of shooting, selects one second external environment image from the plurality of second external environment images, compares the first external environment image and the first external environment feature point information with the selected second external environment image and the second external environment feature point information corresponding to the selected second external environment image, calculates an inter-terminal positional relationship between the information terminal and the other person's information terminal, and outputs meeting support information that guides the user of the information terminal to the terminal position of the other person's information terminal based on the calculated inter-terminal positional relationship.

[0007] According to the present invention, it is possible to provide an information terminal, a meeting support method, and a meeting support system that accurately notify a user of the location of a target location such as a meeting place. Note that problems, configurations, and effects other than those described above will become clear from the description of the following embodiments.

[0008] 1 is a configuration diagram of a meeting support system according to an embodiment of the present invention. FIG. 2 is a hardware configuration diagram of an information terminal. FIG. 3 is a hardware configuration diagram of a server. FIG. 4 is a diagram illustrating an example of usage of the meeting support system according to an embodiment of the present invention. FIG. 5 is a diagram illustrating an image captured from a user information terminal. FIG. 6 is a diagram illustrating an image captured from an other person's information terminal. FIG. 7 is a diagram illustrating an example of external world feature point information. FIG. 8 is a diagram illustrating an example of a correspondence relationship between external world feature points. FIG. 9 is a diagram illustrating an example of measuring coordinate values ​​of external world feature points in a terminal world coordinate system. FIG. 10 is a diagram illustrating an example of evaluating feature quantities of external world feature points. FIG. 11 is a diagram illustrating the contents of a method (1) for acquiring an inter-terminal positional relationship between an other person's information terminal and a user information terminal, and is an explanatory diagram of two directional vectors defined in real space. FIG. 12 is a diagram illustrating the contents of a method (1) for acquiring an inter-terminal positional relationship between an other person's information terminal and a user information terminal, and is a diagram illustrating the relationship between the numerical expressions of two directional vectors in the terminal world coordinate systems of the user information terminal and the other person's information terminal. FIG. 13 is a diagram illustrating a concept of separating the positional relationship between an external world feature point and an HMD 1B, which is the other person's information terminal, into a horizontal projection plane and a vertical projection plane. 13A and 13B are diagrams showing the contents of a method (2) for acquiring the inter-terminal positional relationship of an other information terminal with respect to a user information terminal, and showing a method for calculating coordinate values ​​of the other information terminal in the coordinate system of the user information terminal in a projection plane. FIG. 13B are diagrams showing the contents of a method (2) for acquiring the inter-terminal positional relationship of an other information terminal with respect to a user information terminal, and showing a method for calculating coordinate values ​​of the other information terminal in the coordinate system of the user information terminal in three-dimensional space. FIG. 13C are diagrams showing examples of processing for special situations in the processing of FIG. 13A. FIG. 13C are diagrams showing the contents of a method (2) for acquiring the inter-terminal positional relationship of an other information terminal with respect to a user information terminal, and showing an explanatory diagram of two directional vectors defined in real space. FIG. 13D are diagrams showing an example of changing to a low-distortion shooting mode when shooting an outside world image. FIG. 13D are diagrams showing an example of changing to a wide-angle shooting mode when shooting an outside world image. FIG. 13E are diagrams showing an example of displaying an image shot by a partner terminal when shooting an outside world image. FIG. 13F are diagrams showing an example of displaying a virtual object at a shooting target point when shooting an outside world image. FIG. 13F are diagrams showing an example of displaying a prompt to take additional photos when shooting an outside world image. 10A and 10B are diagrams illustrating an example in which, when capturing an external image, the partner terminal displays a message indicating that additional capture is not possible.10 is a diagram showing an example of information display on a user information terminal (vehicle-mounted terminal 1A) based on terminal position estimation for meeting support. FIG. 11 is a diagram showing an example of information display on an other person's information terminal (HMD 1B) based on terminal position estimation for meeting support. FIG. 12 is a flowchart showing the processing flow of the meeting support system according to the present embodiment. FIG. 13 is a diagram showing an example of setting external feature points in the second embodiment. FIG. 14 is a diagram showing an example of the relationship between a shooting point and a meeting place in the fourth embodiment. FIG. 15 is a diagram explaining the movement of an information terminal in the sixth embodiment. FIG. 16 is a diagram explaining the video matching process in the sixth embodiment. FIG. 17 is a diagram explaining the relationship between a terminal local coordinate system and a terminal world coordinate system. FIG. 18 is a diagram explaining an example of guidance display in the sixth embodiment. FIG. 19 is a diagram explaining another example of guidance display in the sixth embodiment. FIG. 19 is a flowchart showing the processing flow in the sixth embodiment. FIG. 19 is a diagram explaining the processing of variant 1 of the sixth embodiment. FIG. 19 is a diagram explaining the processing of variant 2 of the sixth embodiment.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same components are designated by the same reference numerals throughout the drawings, and redundant explanations will be omitted.

[0010] In this embodiment, the in-vehicle terminal mounted on a vehicle and the portable information terminal carried by a user are collectively referred to as the information terminal. When a user uses a portable information terminal while on a vehicle, it corresponds to the in-vehicle terminal.

[0011] The information terminal according to this embodiment is characterized in that a processor installed in a terminal (hereinafter referred to as an "information terminal") used by a user heading to a meeting place calculates a terminal-to-terminal positional relationship between the information terminal and the other person's information terminal present at a destination point, using information relating to an external world image and external world feature points in the external world image acquired from an information terminal operated by another person (hereinafter referred to as an "other person's information terminal"), and information relating to an external world image captured by a camera installed in the mobile information terminal and external world feature points in the external world image, and outputs guidance information indicating the destination point to the mobile information terminal based on the terminal-to-terminal positional relationship. Therefore, the present invention can increase the commercial value of information processing devices to which the present invention is applied, and is therefore expected to contribute to Goal 8.2 of the Sustainable Development Goals (SDGs) proposed by the United Nations ("SDGs") ("Increase economic productivity through diversification, technological improvement, and innovation, particularly in industries that increase the value of goods and services and in labor-intensive industries").

[0012] The following describes an embodiment of the present invention, exemplifying an embodiment in which the present invention is applied to a meeting support system between a user of an information terminal and another person with whom the user has a meeting. Note that the information terminal may be simply referred to as a terminal. While approaching the meeting place, the information terminal user uses the information terminal to capture an external image of the area near the meeting place from a short distance away. The information terminal then creates an external image feature map showing the relative positional relationship between the external image feature points and the information terminal. Meanwhile, the other person uses the other person's information terminal to capture an external image of the area near the meeting place, generally at the meeting place, and creates external image feature information showing the relative positional relationship between the external image feature points and the other person's information terminal. The external image and external image feature information of both terminals are compared to determine the inter-terminal positional relationship, which is the relative positional relationship between the two terminals. The information terminal then guides the user to the meeting place based on this inter-terminal positional relationship information. In this case, the meeting place corresponds to the destination point to which the user of the information terminal is heading. The destination point is stationary in the world coordinate system. The user moves toward the destination point together with the information terminal.

[0013] First Embodiment FIG. 1 is a configuration diagram of a meeting support system 100 according to this embodiment.

[0014] An example of an information terminal is an in-vehicle terminal 1A. While approaching the meeting place, the in-vehicle terminal 1A captures an external image of the vicinity of the meeting place and generates external feature point information indicating the relative positional relationship between the external feature points in the external image and the information terminal. The other person waiting at the meeting place carries an HMD 1B or a smartphone 1C, which is an information terminal of the other person.

[0015] The in-vehicle terminal 1A, the HMD 1B, and the smartphone 1C are connected to one another via a network 3. Furthermore, the in-vehicle terminal 1A, the HMD 1B, and the smartphone 1C are connected to a server 2 via the network 3. The server 2 is a server that provides a meeting support service, and may be a cloud server or an edge server.

[0016] The other person uses the other person's information terminal to capture an external world image near the meeting place, creates external world feature point information indicating the relative positional relationship between the external world feature points in the external world image and the other person's information terminal, and transmits this information to the vehicle-mounted terminal 1A. The vehicle-mounted terminal 1A determines the other person's position at the meeting place using the received external world image and external world feature point information, the external world image captured by the vehicle-mounted terminal 1A, and the created external world feature point information.

[0017] The in-vehicle terminal 1A includes a processor connected to in-vehicle equipment such as an in-vehicle camera, a car navigation system, and a GPS mounted on the vehicle 1. For example, the in-vehicle terminal 1A may be configured by connecting a smartphone operated by a passenger in the vehicle 1, such as the driver, to in-vehicle equipment such as an in-vehicle camera using Bluetooth (registered trademark). The in-vehicle terminal 1A may also be configured by connecting the in-vehicle equipment and the processor via a vehicle network (CAN). Furthermore, the in-vehicle terminal 1A may be configured by a standalone mobile information terminal such as a smartphone operated by a passenger in the vehicle 1.

[0018] The information terminal may be an HMD 1B or a smartphone 1C. That is, the information terminal refers to a terminal carried by a user who is moving toward a target location, which is a meeting place, and the type of device used may be the in-vehicle terminal 1A, the HMD 1B, or the smartphone 1C.

[0019] Similarly, the other person's information terminal means a terminal for specifying a target point, and is not limited to the HMD 1B or the smartphone 1C, but may also be the in-vehicle terminal 1A.

[0020] FIG. 2 is a diagram showing the hardware configuration of the information terminal.

[0021] 2 shows the components necessary to realize the functions of the information terminal according to this embodiment and the components that are often typically installed. Furthermore, since the configurations of the information terminals used in this embodiment are similar to each other, they are described together. The configurations of the in-vehicle terminal 1A, HMD 1B, and smartphone 1C are distinguished by reference numerals A, B, and C, respectively. The in-vehicle terminal 1A, HMD 1B, and smartphone 1C may be collectively referred to as the terminal.

[0022] The in-vehicle terminal 1A, the HMD 1B, and the smartphone 1C include outer cameras 111A, 111B, and 111C, distance measurement sensors 113A, 113B, and 113C, position and orientation sensors 135A, 135B, and 135C, geomagnetic sensors 117A, 117B, and 117C, positioning sensors 118A, 118B, and 118C, and a real-time clock (RTC) The components include clocks 114A, 114B, and 114C, displays 119A, 119B, and 119C, microphones 121A, 121B, and 121C, speakers 122A, 122B, and 122C, processors 125A, 125B, and 125C, memories 128A, 128B, and 128C, and network communicators 120A, 120B, and 120C, which are connected to one another via buses 140A, 140B, and 140C that connect the components together.

[0023] Specific examples of the distance measurement sensors 113A, 113B, and 113C include a TOF (Time of Flight) sensor and a LiDAR (Laser Imaging Detection and Ranging) sensor.

[0024] The network communicators 120A, 120B, 120C are connected to antennas 123A, 123B, 123C that transmit and receive network communication signals.

[0025] The displays 119A, 119B, and 119C and the speakers 122A, 122B, and 122C correspond to output devices, and the microphones 121A, 121B, and 121C correspond to input devices.

[0026] The position and orientation sensors 135A, 135B, and 135C are used to measure the position and orientation of the terminal. Here, orientation refers to the orientation of the terminal in the real space in which the terminal exists. Specific configuration examples of the position and orientation sensors 135A, 135B, and 135C may include position posture cameras 136A, 136B, and 136C (hereinafter abbreviated as PP cameras), position posture distance measurement sensors 137A, 137B, and 137C (hereinafter abbreviated as PP distance measurement sensors), acceleration sensors 115A, 115B, and 115C, and gyro sensors 116A, 116B, and 116C.

[0027] The PP cameras 136A, 136B, and 136C may be the same as the outer cameras 111A, 111B, and 111C.

[0028] Furthermore, the PP distance measuring sensors 137A, 137B, and 137C may be the same as the distance measuring sensors 113A, 113B, and 113C.

[0029] Feature points that exist in the external world (hereinafter referred to as "external world feature points") are used to measure the position and orientation using the PP cameras 136A, 136B, and 136C and the PP ranging sensors 137A, 137B, and 137C. A plurality of external world feature points are used, and for example, external world feature points inside a room or external world feature points on a building outdoors may be used. As a preparation, the coordinate values ​​of the external world feature points are determined from the direction and distance measurements of the external world feature points. If the coordinate values ​​of the external world feature points in the external world coordinate system are known, this preparation can be omitted.

[0030] Then, by measuring changes in the direction and distance of the external feature points as seen from the terminal, changes in the position and orientation of the terminal from the relative positional relationship with the initial external feature points are measured. When measuring distance using a stereo camera method, only PP cameras 136A, 136B, and 136C may be used.

[0031] Note that while grasping the vertical direction is important in practical applications, if the coordinate values ​​of the external feature points in the external coordinate system (world coordinate system) and the vertical direction in the external coordinate system are known, the relative positional relationship between the terminal housing attitude and the vertical direction can be grasped by measuring the external feature points using PP cameras 136A, 136B, and 136C and PP ranging sensors 137A, 137B, and 137C. That is, for example, by measuring the coordinate values ​​of three external feature points in the terminal local coordinate system, the relationship between the numerical representations of two directional vectors between the external feature points in the external coordinate system and the terminal local coordinate system can be determined, and a rotation transformation that aligns the orientation of the terminal local coordinate system with the external coordinate system can be obtained. This determines the relationship between the terminal housing and the vertical direction.

[0032] Measurement of position and attitude using acceleration sensors 115A, 115B, and 115C and gyro sensors 116A, 116B, and 116C is performed by integrating acceleration and angular acceleration. This allows for determination of displacement from a reference position and reference attitude. Furthermore, measurement of the gravitational acceleration vector by acceleration sensors 115A, 115B, and 115C also allows for determination of the terminal's housing attitude and its relative position in the vertical direction.

[0033] The position and attitude of the terminal may be measured by referring to both the measurements using the PP cameras 136A, 136B, 136C and the PP ranging sensors 137A, 137B, 137C and the measurements using the acceleration sensors 115A, 115B, 115C and the gyro sensors 116A, 116B, 116C, or by using measurements from other types of sensors.

[0034] Positioning sensors 118A, 118B, and 118C are used to measure the indoor and outdoor positions of terminals. Specific examples of indoor positioning sensors include those that use beacon signals and location indicator marks. Outdoor positioning sensors include, for example, GPS (Global Positioning System) receivers that use signals from satellites.

[0035] The network communicators 120A, 120B, and 120C are communication interfaces that perform wireless communication between at least the terminal and the access point 4 via short-range wireless communication, wireless LAN, or base station communication, and include communication processing circuits corresponding to various predetermined communication interfaces and are connected to the antennas 123A, 123B, and 123C. The network communicators 120A, 120B, and 120C transmit and receive biometric information, video data, control signals, and the like. Note that short-range wireless communication is performed using a wireless LAN such as Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or Wi-Fi (registered trademark). Furthermore, for base station communication, long-distance wireless communication such as LTE (Long Term Evolution, registered trademark), LTE-Advanced, Mobile WiMAX (Worldwide Interoperability for Microwave Access, registered trademark), or WiMAX2 may be used.

[0036] Displays 119A, 119B, and 119C are installed in a manner that allows the terminal users to view images. Processors 125A, 125B, and 125C control the operation of the terminals.

[0037] The processors 125A, 125B, and 125C are configured by, for example, a CPU.

[0038] The memories 128A, 128B, and 128C are configured by flash memory and nonvolatile memory. The memories 128A, 128B, and 128C also store programs 126A, 126B, and 126C such as an OS (Operating System) and operation control applications, and data 127A, 127B, and 127C used by the processors 125A, 125B, and 125C.

[0039] The processors 125A, 125B, 125C load the programs 126A, 126B, 126C into the memories 128A, 128B, 128C and execute them, and read out the data 127A, 127B, 127C as needed and use it in the execution processing of the programs 126A, 126B, 126C.

[0040] The smartphone 1C further includes, as components, an inner camera 112C, a touch panel 124 as an input interface, and a telephone network communication device 131C.

[0041] FIG. 3 is a diagram showing the hardware configuration of the server.

[0042] In the server 2, a processor 225, a memory 228, and a network communicator 220 are connected to one another via a bus 240 that connects the various components. The network communicator 220 is connected to an antenna 223 that transmits and receives network communication signals.

[0043] The processor 225 controls the operation of the server and is configured by, for example, a CPU.

[0044] The memory 228 is configured by a flash memory and a nonvolatile memory, and also stores programs 226 such as an OS (Operating System) and an operation control application, and data 227 used by the processor 225.

[0045] The processor 225 loads the program 226 into the memory 228 and executes it, and reads out the data 227 as necessary to use it in the execution process of the program 226 .

[0046] [Processing Flow] FIG. 4 is a diagram showing an example of the usage status of the meeting support system 100 according to this embodiment.

[0047] Assume that a passenger and a taxi are meeting at the location where the passenger is standing in the figure. The passenger corresponds to a third party who specifies the destination point, which is the meeting place. The taxi corresponds to vehicle 1 in Figure 1, and the taxi driver corresponds to a user traveling toward the destination point, which is the meeting place.

[0048] The taxi is equipped with an in-vehicle terminal 1A, and the passenger carries an HMD 1B. The in-vehicle terminal 1A corresponds to the user information terminal. The HMD 1B corresponds to the other person's information terminal. The following process is basically the same even if the other person's information terminal is a smartphone 1C. To know the relative positional relationship between the terminals, a video of the vicinity of the meeting place is taken. The video may be a video or a still image. The relative positional relationship between the terminals may be referred to as the inter-terminal positional relationship.

[0049] Fig. 5 is a diagram showing an image captured from a user information terminal, and Fig. 6 is a diagram showing an image captured from another person's information terminal. The first external world image 1000 shown in Fig. 5 includes first external world feature points 1010A, 1010B, and 1010C. Similarly, the second external world image 1001 shown in Fig. 6 includes second external world feature points 1011A, 1011B, and 1011C. Because it is necessary for both terminals to capture roughly the same location, approximate data on the subject position, shooting location, and shooting direction may be communicated from one terminal to the other terminal or bidirectionally using positioning data from a general-purpose positioning system such as GPS that can determine the approximate position.

[0050] (Method for acquiring the positional relationship between terminals (1)) First, we will explain the terms related to coordinate systems. First, a "world coordinate system" is a coordinate system that is stationary relative to the external world, and a "local coordinate system" is a coordinate system that is not stationary relative to the external world but is fixed to some kind of object. Then, a "terminal world coordinate system" is a coordinate system that is fixed relative to the external world and is used by the terminal under consideration, and a "terminal local coordinate system" is a coordinate system that is fixed to the housing of the terminal under consideration.

[0051] Each terminal requires some kind of "world coordinate system" to describe its own position and orientation relative to the external world as its control data. The "world coordinate system" used by each terminal is called the "terminal world coordinate system." Generally, the "terminal world coordinate system" differs from terminal to terminal. The core of the present invention is a technology described in "Methods (1) and (2) for Acquiring Positional Relationship Between Terminals," which compares external world images captured by two terminals and determines coordinate transformation parameters between the two terminal world coordinate systems. These coordinate transformation parameters determine the position coordinates of the other terminal within the terminal world coordinate system, or vice versa. The difference between acquisition methods (1) and (2) is the accompanying information of the external world image. In using external world images, the present invention falls under the category of VPS (Visual Positioning System) technology. However, the present invention differs from conventional VPS technology in that the amount of image information required to meet the objectives of each embodiment is small.

[0052] The "terminal local coordinate system" is used to describe the positions of external feature points and the direction as seen from the terminal, based on the terminal's position and orientation.

[0053] In the explanation, we sometimes say "numerical expression in a coordinate system." We also sometimes shorten it to "expression in a coordinate system," but in this case "expression" means "numerical expression." Specifically, when we say "numerical expression" of a point or position in the outside world, we are referring to the coordinate value of the point or position in a certain coordinate system. If we express it in terms of components for each coordinate axis in a three-dimensional coordinate system, it is as follows: (P Ｘ , P Ｙ , P Ｚ ) is data such as this. Direction, or the "numerical expression" of a directional vector, which is the same thing, is a numerical representation of a directional vector in a certain coordinate system. If expressed in components for each coordinate axis in a three-dimensional coordinate system, it is expressed as (V Ｘ , V Ｙ , V Ｚ) The specific values ​​of these "numerical representations" change when the coordinate system changes, even if they represent the same point or the same directional vector. In the process described below, the specific values ​​of the "numerical representation" of the same point or the same directional vector change depending on the respective "terminal world coordinate systems," so conversion parameters are found to align the "numerical representation" between different "terminal world coordinate systems" based on these differences. Knowing the position of the other terminal means knowing the "numerical representation" of the other terminal's position in the "terminal world coordinate system" of the own terminal.

[0054] Next, we will explain the symbols. In the following explanation, unless otherwise specified, when rotation, direction vectors, and vectors (including point position indications) are represented by a single symbol, they will be represented as quaternions. In addition, the norm of both rotation and direction vectors is normalized to 1. Furthermore, when a direction vector or vector is represented by a quaternion, the real part of the quaternion is set to 0. We will first organize the symbols used in the explanation.

[0055] First, the basis elements of a quaternion other than 1 are represented as i, j, and k, and unless otherwise specified, when representing a direction vector or vector, the relationship to the axis directions of the coordinate system under consideration is X axis: i, Y axis: j, and Z axis: k. For example, when the basis element i in quaternion notation represents a vector, this vector means (1, 0, 0) in the component notation for each of the coordinate axes X, Y, and Z. In each three-dimensional coordinate system, the coordinate axes are represented by a combination of the letters X, Y, and Z and their subscripts.

[0056] A quaternion Q that rotates a direction vector n by an angle λ in the direction of a right-handed screw. Ｒ (n, λ) is expressed by the following formula: Q Ｒ (n, λ) = cos(λ / 2) + sin(λ / 2)n (1) Note that rotating in the direction in which a right-handed screw advances means rotating in the direction in which the screw rotates when a right-handed screw advances (clockwise when looking in the direction in which the screw advances).

[0057] Here, Q ＊ is the conjugate quaternion of Q, the order of multiplication when vector a is rotated to vector b by rotation Q is as follows: b = QaQ ＊ ... (2)

[0058] Specifically, the quaternion that rotates 90 degrees around the direction vector n in the direction of n, in the direction of a right-handed screw, is expressed as H(n). H(n) is expressed as follows: H(n) = 2 －１／２ +2 －１／２ n... (3)

[0059] [·] is an operator that normalizes a non-zero quaternion to norm 1. If q is any non-zero quaternion, [·] is defined as follows: [q] = q(qq ＊ ) －１／２ ...(4)

[0060] P Ｔ (n) is an operator that projects a vector onto a plane perpendicular to the direction vector n. n is assumed to be normalized. The projection of vector p is expressed by the following equation: P Ｔ (n) p = p + nS Ｃ (np) (5) Here, Sc(q) denotes the real part of the quaternion q.

[0061] Together with the above symbols, the direction vector n １ and direction vector n ２ Rotate around an axis perpendicular to the plane containing n １ to n ２ Rotation Q to be superimposed １２ Use equation (6) to find Q １２ = [1-n ２ n １ ]・・・(6)

[0062] It can be easily confirmed that the formula (6) holds true by directly calculating the right-hand side, taking care to set the real part of the direction vector to 0.

[0063] Now, in order to match the external feature points in the external image of the area around the meeting point taken by each device, external feature point information for each image is created. The image can be a still image or a video. In the case of a video, at least one frame of the image is extracted and used.

[0064] The external world feature point information is list information that includes the positions of the external world feature points in the video and coordinate value data in the terminal's world coordinate system, corresponding to the captured video. An example of the external world feature point information is shown in FIG. 7A. The coordinate values ​​in FIG. 7A are written in normal component notation. Such external world feature point information is created for each of the videos from both terminals. The coordinate values ​​of the matched external world feature points may be obtained later using the feature point information or external world feature point matching, which will be described later.

[0065] When using the method (2) for acquiring the positional relationship between terminals described later, one of the terminals uses the value of a direction vector in the terminal world coordinate system pointing from the terminal to the external world feature point in place of the coordinate value of the external world feature point in the external world feature point information.

[0066] Here, the position on the video refers to coordinates on the video, for example, coordinates with the upper left corner of the video as the origin, the horizontal direction with the right direction as the positive direction as the U coordinate, and the vertical direction with the downward direction as the positive direction as the V coordinate ( FIG. 8 ). In the following, the coordinate system refers to a world coordinate system fixed to the external world unless otherwise specified. The external world feature information also includes the feature amounts of the external world feature points, which will be described later. Using these feature amounts as clues, external world feature point matching, which will be described later, is performed to determine the correspondence between the external world feature points in the video of both devices. An example of an external world feature point matching table that summarizes this correspondence is shown in FIG. 7B .

[0067] First, as a method (1) for acquiring the positional relationship between terminals, a case will be described in which the external world feature point information of both terminals contains coordinate value data of the external world feature point. To obtain the coordinate value of the external world feature point in the terminal world coordinate system, first, the outer cameras 111A and 111B measure the direction vector of the external world feature point as seen from the terminal position relative to the terminal housing, and the ranging sensors 113A and 113B measure the distance from the terminal position to the external world feature point to obtain the coordinate value in the terminal local coordinate system. Next, the position and orientation sensors 135A and 135B measure the position and orientation of the terminal in the world coordinate system, and the coordinate value in the terminal local coordinate system is converted to a coordinate value in the terminal world coordinate system, thereby obtaining the distance. Distance measurement may also be performed using a moving stereo method using the outer cameras 111A and 111B.

[0068] Here, the method for calculating the coordinate values ​​of the external feature points will be described in detail. First, the terminal local coordinate system {X Ｌ Axis, Y Ｌ axis, Z Ｌ Axis, origin: O Ｌ} is defined (Fig. 8). Then, the unit vectors in the directions of the coordinate axes are defined as N ＸＬ , N ＹＬ , N ＺＬ The specific notation in quaternions is N ＸＬ = i, N ＹＬ = j, N ＺＬ = k, where i, j, and k are the basis elements of the quaternion. The origin of the terminal local coordinate system is the point where the direction of the incident light from the outside world converges. The optical axis direction is X Ｌ axis, and the horizontal direction of the image is perpendicular to the optical axis and points leftward in the Y direction. Ｌ Axis, vertical direction of the image is Z Ｌ The axis is X. Ｌ The axis passes through the center of the image. Ｗ Axis, Y Ｗ axis, Z Ｗ Axis, origin: O Ｗ} is defined to recognize the position and orientation of the terminal. As an example of the definition of the terminal world coordinate system, there is a method of fixing the terminal local coordinate system at the time of terminal initialization to the external world as it is and using it as the terminal world coordinate system. The maximum value of the video coordinate value is (U Ｍ , V Ｍ ) and the horizontal angle of view is μ Ｕ , vertical angle of view is μ Ｖ , the image plane is perpendicular to the optical axis, and distortion caused by the optical system is also corrected.

[0069] Now, the position of the external feature point on the image (U Ｐ , V Ｐ ) The origin of the terminal local coordinate system O Ｌ The distance from the center of the image to the center of the image is f, and the coordinate value on the image (U Ｐ , V Ｐ ) is the position of the point on the image represented by P Ｎ Then, P Ｎ The three-dimensional expression of P in the terminal local coordinate system is expressed by the following equation: Ｎ = fN ＸＬ +fS Ｎ ... (7)

[0070] Here, S Ｎ is expressed as follows in three-dimensional terms in the terminal local coordinate system: Ｎ = ((U Ｍ -2U Ｐ ) / 2U Ｍ ) tan(μ Ｕ / 2) N ＹＬ +((V Ｍ -2V Ｐ ) / 2V Ｍ ) tan(μ Ｖ / 2) N ＺＬ ...(8)

[0071] From this, the origin of the terminal local coordinate system O Ｌ The external feature point direction vector N Ｒ is given by the following equation in numerical expression in the terminal local coordinate system: Ｒ = [N ＸＬ +S Ｎ ]・・・(9)

[0072] The external feature point direction vector N Ｒ and the distance d from the terminal position to the external feature point, which is calculated separately. Ｒ multiplied by , the numerical representation R of the coordinates of the external feature points in the terminal local coordinate system is obtained. Ｌ is obtained. R Ｌ = d Ｒ N Ｒ ...(10)

[0073] Representation of external feature point coordinate values ​​in the device local coordinate system R Ｌ From the expression R in the terminal world coordinate system Ｗ To convert to this, information on the orientation and origin position of the terminal local coordinate system in the terminal world coordinate system is used. The orientation and origin position of the terminal local coordinate system are known because the changes from the initialized state are measured by the position and orientation sensors 135A and 135B. The rotation of the terminal local coordinate system in the terminal world coordinate system from the state where the directions of the X axes, Y axes, and Z axes are aligned to the current point in time is expressed as Q. ＬＷ The coordinate value of the origin of the terminal local coordinates in the terminal world coordinate system is O ＬＷ Then, R Ｗ is calculated as follows: R Ｗ = QＬＷ R Ｌ Q ＬＷ ＊ +O ＬＷ ...(11)

[0074] In the method (2) for acquiring the positional relationship between terminals, which will be described later, the external feature point information of one terminal uses a direction vector from the terminal position to the external feature point instead of the coordinate value of the external feature point. The numerical expression in the terminal world coordinate system of the direction vector from the terminal position to the external feature point is Ｗ Then, N Ｗ is calculated as follows: N Ｗ = Q ＬＷ N Ｌ Q ＬＷ ＊ ...(11')

[0075] Examples of external field feature points are shown in Figures 5 and 6. Here, the corners of window frames of three buildings are extracted as external field feature points as examples of stationary objects. To determine the positional relationship between terminals, three external field feature points that are not on the same line are required. Since there is no guarantee that individual external field feature points can be matched between external field images captured by both terminals with 100% probability, it is desirable to list three or more external field feature points in the external field feature point information. In Figures 5 and 6, the matched external field feature points are shown as first external field feature points 1010A, 1010B, and 1010C and second external field feature points 1011A, 1011B, and 1011C.

[0076] 5 is an example of an external world image captured by a user information terminal, which is designated as a first external world image 1000, and FIG. 6 is an example of an external world image captured by another person's information terminal, which is distinguished from a second external world image 1001. First external world feature points 1010A, 1010B, and 1010C are captured in the first external world image 1000, and second external world feature points 1011A, 1011B, and 1011C are captured in the second external world image 1001. Here, for example, an external world feature point captured in the first external world image is referred to as a first external world feature point, but if there is no need to specify in which image the referenced external world feature point was captured, the expressions specifying the image, such as first or second, may be omitted.

[0077] The distinction between first and second coordinate systems for the external world image may also be used for the terminal coordinate system. That is, the world coordinate system used by the user information terminal may be referred to as the first terminal world coordinate system, and the world coordinate system used by the other person's information terminal may be referred to as the second terminal world coordinate system.

[0078] Here, a method for matching external feature points in the external image of both terminals will be described. First, external feature points, mainly corners such as window frames of buildings, are extracted from the external image. Next, feature quantities of the extracted external feature points are calculated from the external image. There are various methods for calculating feature quantities. For example, SIFT (Scale-Invariant Feature Transform), SURF (Speed-Up Robust Features), BRIEF (Binary Robust Independent Elementary Features), etc., and any feature quantity may be used for matching.

[0079] Here, an overview of SIFT features is explained (Fig. 9). First, the maximum gradient direction of the brightness gradient near the external feature point is found, and the external image area is divided into 16 4x4 blocks near the feature point according to this direction. Then, the brightness gradient in each block is represented as a histogram in eight directions. The data of this histogram is collected and represented as a 128-dimensional vector value, which is the SIFT feature.

[0080] Next, an example of a procedure for matching feature points based on feature amounts will be described. While Figs. 5 and 6 show first and second external world feature points whose correspondence relationship is determined as a result of the matching process, it is not clear which first and second external world feature points represent the same external world feature point until the result of the matching process is obtained. Therefore, in the following description of the processing procedure, the labels of the first and second external world feature points are independent. The label of the first external world feature point during the matching process is referred to as a ｈ (h=1, 2, 3...M), and the label of the second external feature point is b ｈ (h=1, 2, 3...N). The first external feature point a r1 In order to find the second external feature point corresponding to the first external feature point a r1The distance between the feature vectors of the second external feature point b is calculated. Here, the distance between the feature vectors is the norm of the difference between the two feature vectors. Then, the second external feature point b, which has the smallest inter-feature distance, is calculated. r1 the first external feature point a r1 is the corresponding second external feature point, and matching is established. However, if this smallest inter-feature distance is 0.8 times or more the second smallest inter-feature distance, it is determined that there is no corresponding second external feature point, and matching is not established. This is because the existence of another second external feature point with a close inter-feature distance may indicate an incorrect correspondence.

[0081] After the first stage of matching, a cross-check is performed. First, in the first stage of matching, the first external feature point a r1 The second external feature point b r1 Next, the second external feature point b r1 is fixed, and an external feature point is selected from the first external feature point by the above matching method. If there is no first external feature point that matches in this reverse matching, the first external feature point a r1 and the second external feature point b r1 If there is a first external feature point for which matching is established in the reverse direction, the first external feature point is set as a r２ This first external feature point a r２ is the first external feature point a r１ If it is equal to, the first external feature point a r１ and the second external feature point b r１ If the first external feature point a is not equal to the first external feature point a, the first external feature point a is considered to be equal to the first external feature point a. r１ and the second external feature point b r１ is determined to be unsuccessful. Note that the first external world feature point and the second external world feature point for which matching has been achieved may be managed by assigning the same label. In the following description, the external world feature point α, the external world feature point β, and the external world feature point γ will be used as examples of the external world feature points for which matching has been achieved in a pair of feature points in the set of first external world feature points and feature points in the set of second external world feature points.

[0082] If the number of matching external feature points is less than a predetermined number, the external areas captured by the two terminals may be different, so the video may be re-captured.

[0083] The extraction of external world feature points, the calculation of feature amounts, and the execution of matching may be performed by either terminal. The extraction of external world feature points and the calculation of feature amounts may be performed by each terminal, and the calculation results of feature amounts may be collected on one terminal to execute matching. Alternatively, external world video information may be sent from one terminal to the other, and the subsequent processing up to matching may be executed on one terminal. Furthermore, the calculation processing may be performed by server 2.

[0084] Furthermore, the measurement of the coordinate values ​​of the external world feature points may be performed for all the extracted external world feature points in the external world image, or may be limited to some of the external world feature points required for subsequent processing. When the terminal is moving, in order to obtain the coordinate values ​​of the external world feature points after the timing of capturing the external world image used for external world feature point matching, it is sufficient to capture the external world image from different capturing positions, measure the distance to the external world feature point using a moving stereo method, and obtain the coordinate values.

[0085] From among the external feature points for which matching has been established between the external image of both terminals in the processing up to this point, three external feature points that are not on the same line are selected, and a process is performed to determine the relative positional relationship between the two terminals.

[0086] FIG. 10 is a diagram showing the contents of a method (1) for acquiring the inter-terminal positional relationship of another person's information terminal with respect to a user information terminal, and is an explanatory diagram of two direction vectors defined in real space.

[0087] As shown in FIG. 10, N １ , N ２These two direction vectors can be defined, and the rotation for matching the orientations of the two terminal world coordinate systems can be calculated from the numerical representation of these two direction vectors in the terminal world coordinate systems of the in-vehicle terminal 1A and the HMD 1B. Also, the difference in the origins of the two terminal world coordinate systems can be calculated from the numerical representation of any one point in real space in the terminal world coordinate systems. Information on this rotation for matching the orientations and the difference in origins is conversion information between the terminal world coordinate systems of the in-vehicle terminal 1A and the HMD 1B. If conversion between coordinate systems can be performed, it is naturally possible to know the position of the other terminal.

[0088] In addition, N in the real space １ , N ２ If the direction vector can be measured and recognized by another method, the number of external feature points to be used can be reduced, and this method will be described later as a modified example.

[0089] FIG. 11 is a diagram showing the contents of a method (1) for acquiring the inter-terminal positional relationship of an other information terminal with respect to a user information terminal, and is a diagram showing two direction vectors in the terminal world coordinate systems of the user information terminal and the other information terminal.

[0090] Direction vector N １ , N ２ The numerical expression in the terminal world coordinate system of the vehicle-mounted terminal 1A is N （Ａ） １ , N （Ａ） ２ and the numerical expression in the terminal world coordinate system of the HMD 1B is N （Ｂ） １ , N （Ｂ） ２ These direction vectors are shown in Fig. 11. Here, the space in Fig. 11 is the coordinate space of the vehicle-mounted terminal 1A, and N （Ｂ） １ , N （Ｂ） ２ When expressed in numerical terms in the coordinate system of the vehicle-mounted terminal 1A, N （Ａ） １ , N （Ａ） ２ The direction vector is different from that of （Ｂ） １ , N （Ｂ） ２ Rotate to N （Ａ）１ , N （Ａ） ２ This explains the transformation that is superimposed on

[0091] First, N （Ａ） １ and N （Ｂ） １ Rotate around an axis perpendicular to the plane containing N （Ｂ） １ N （Ａ） １ Such a rotation Q １ is expressed by the following equation (12): １ = [1-N （Ａ） １ N （Ｂ） １ ]・・・(12)

[0092] In addition, from the beginning, N （Ｂ） １ N （Ａ） １ In the special case where １ = 1, i.e. no rotation.

[0093] After this rotation, N （Ｂ） ２ The direction vector of N （Ｂ）＃ ２ If you write this, （Ｂ）＃ ２ is expressed by the following equation (13): （Ｂ）＃ ２ =Q １ N （Ｂ） ２ Q １ ＊ ...(13)

[0094] Next, N （Ｂ）＃ ２ N （Ａ） １ Rotate around the axis and （Ａ） ２ The rotation at this time is Q ２ When written as Q ２ is expressed as the following equation (14): Q ２ = [1-[P Ｔ (N （Ａ） １ ) N （Ａ） ２ ][P Ｔ (N（Ａ） １ ) N （Ｂ）＃ ２ ]]...(14) where P Ｔ (n) is an operator that projects a vector onto a plane perpendicular to the direction vector n (defined in equation (5)). （Ｂ）＃ ２ N （Ａ） ２ In the special case where ２ = 1, i.e. no rotation.

[0095] From the above, the numerical expression of the direction vector expressed in the terminal world coordinate system of HMD1B is rotation (Q ２ Q １ ) can be converted into a numerical expression of a direction vector expressed in the terminal world coordinate system of the vehicle-mounted terminal 1A. For the following explanation, rotation Q ＢＡ is defined by the following equation (15): ＢＡ =Q ２ Q １ ...(15)

[0096] Next, the difference between the origins of the two terminal world coordinate systems is calculated. ＢＡ is expressed by the following equation (16): ＢＡ =D ＃（Ａ） -Q ＢＡ D ＃（Ｂ） Q ＢＡ ＊ ...(16)

[0097] Here, D ＃（Ａ） , D ＃（Ｂ） are numerical representations of arbitrarily selected external points in the terminal world coordinate system of the vehicle-mounted terminal 1A and the terminal world coordinate system of the HMD 1B. For example, if an external feature point α is selected and D ＃（Ａ） , D ＃（Ｂ） If you substitute specific values ​​for , ＢＡ is obtained. In this way, O ＢＡ If we use the above formula, the conversion of numerical representation between coordinate systems of any point is given by the following formula (17): （Ａ） =Q ＢＡ D （Ｂ） Q ＢＡ ＊ +O ＢＡ...(17)

[0098] The inverse transformation is given by the following equation (18): D （Ｂ） = Q ＢＡ ＊ D （Ａ） Q ＢＡ -Q ＢＡ ＊ O ＢＡ Q ＢＡ ...(18)

[0099] Here, D （Ａ） , D （Ｂ） are numerical representations of an arbitrary point in the coordinate system of the vehicle-mounted terminal 1A and the coordinate system of the HMD 1B, respectively.

[0100] The rotation Q, which is the transformation parameter between the coordinate systems of both terminals, is calculated as follows: ＢＡ and the difference between the origin position O ＢＡ By sharing this information between both terminals, each terminal can obtain the position coordinates of the other terminal in its own coordinate system and convert them into coordinates in its own coordinate system. In other words, both terminals can grasp the positional relationship between the terminals. Alternatively, one terminal may perform the coordinate conversion calculation and share the calculation results.

[0101] Note that, by using the above method, when the coordinate values ​​of the external world feature points in the terminal world coordinate system are known, it is possible to determine the position and orientation of the terminal with respect to the terminal world coordinate system using PP cameras 136A, 136B, and 136C and PP ranging sensors 137A, 137B, and 137C. Since the external world feature points can be identified using PP cameras 136A, 136B, and 136C and the coordinate values ​​of the external world feature points in the terminal local coordinate system can be measured using PP ranging sensors 137A, 137B, and 137C, it is possible to determine the transformation parameters between the terminal world coordinate system and the terminal local coordinate system using the above method.

[0102] (Variation of Method (1) for Acquiring Positional Relationship Between Terminals) If both terminals can recognize a specific direction that allows them to recognize a direction, rather than a direction between points in real space, the number of required external feature points can be reduced.

[0103] For example, the vertically upward direction can be recognized by measuring the direction of gravitational acceleration using acceleration sensors 115A and 115B on both terminals. If this vertically upward direction is considered as one of the directions in the real space described above, the number of external feature points used can be two. In other words, the direction vector from external feature point α to β is N １ and the vertical upward direction vector is N ２ If so, the transformation formula for the coordinate systems between the two terminals can be found using the same calculation procedure.

[0104] Furthermore, as another specific direction in the real space, for example, the north direction is measured by the geomagnetic sensors 117A and 117B, and a direction vector N １ In order to find the difference between the origins of both coordinate systems using equation (16), the coordinate values ​​of one point in the external world must be known in both coordinate systems, so one external feature point is used.

[0105] The specific direction in real space is not limited to the above, and other directions may be used. For example, the direction of a landmark that is sufficiently far from the terminal may be used as the direction in real space.

[0106] As described above, by using real space directions other than the directions between external field feature points, it is possible to reduce the number of external field feature points used and the calculation load for external field feature point matching.

[0107] (Method (2) of acquiring positional relationship between terminals) Next, a processing method will be described when the coordinate values ​​of external world feature points are measured only on one terminal. The other terminal measures only the direction of the external world feature points. Here, it is assumed that the coordinate values ​​of the external world feature points are measured on the vehicle-mounted terminal 1A, but it is the same even if the coordinate values ​​of the external world feature points are measured on the HMD 1B. Note that when distance measurement is performed using a moving stereo technique or when using a processing procedure that does not measure distance as in this method (2) of acquiring positional relationship between terminals, a hardware configuration that does not use distance measurement sensors 113A, 113B and PP distance measurement sensors 137A, 137B is also possible.

[0108] The vehicle-mounted terminal 1A generates external world feature point information based on coordinate value data of the external world feature points, and the HMD 1B generates external world feature point information based on directional vector data to the external world feature points. The external world feature point matching is the same as the method used in the method (1) for acquiring the positional relationship between terminals.

[0109] 12 is a diagram showing the concept of separating the positional relationship between the matched external feature points and the HMD 1B, which is the other person's information terminal, into a horizontal projection plane and a vertical projection plane. A method for determining the vertical projection plane will be described later.

[0110] First, as shown in Fig. 12, the positional relationship between the external feature points and the HMD 1B, which is the other person's information terminal, is orthogonally projected onto a horizontal projection plane and a vertical projection plane, and the positional relationship within the horizontal plane and the positional relationship within the vertical plane are considered separately. The vertical direction required for projection onto the horizontal projection plane can be recognized by both the in-vehicle terminal 1A and the HMD 1B, and is performed by measuring the direction of gravitational acceleration using acceleration sensors 115A and 115B, for example.

[0111] 12, the terminal position X of the HMD 1B and the positions of the external world feature points α, β, and γ in the real space projected onto a horizontal projection plane are represented by terminal position X' and external world feature points α', β', and γ'. Also, the terminal position X of the HMD 1B and the positions of the external world feature points α, β, and γ projected onto a vertical projection plane are represented by terminal position X'' and external world feature points α'', β'', and γ''.

[0112] When a taxi approaches a passenger, the difference in vertical position between the taxi and the passenger is usually not large enough to cause inconvenience during the meeting. Therefore, if the relative positional relationship between the two in the horizontal projection plane is known, support for the meeting place can be provided, but the processing for a general case will also be explained. First, the processing content of the in-vehicle terminal 1A estimating the terminal position X' of the other person's information terminal in the horizontal projection plane will be explained.

[0113] FIG. 13 is a diagram showing the contents of a method (2) for acquiring the inter-terminal positional relationship between a user information terminal and another person's information terminal.

[0114] The coordinate values ​​of the external feature points α, β, and γ in the terminal world coordinate system of the vehicle-mounted terminal 1A are measured by the vehicle-mounted terminal 1A.

[0115] On the other hand, in the HMD 1B, the direction vectors from the terminal to the external feature points α, β, and γ are expressed numerically in the coordinate system of the HMD 1B, N α , N β , N γ Ask for.

[0116] Direction vector N to the external feature point α , N β , N γ Once this is determined, the vertical upward direction vector is N Ｖ and the angle of view (θ') between the external feature points as seen from the position of the HMD 1B (terminal position X') in the horizontal projection plane. αβ , θ' βγ , θ' γα ) is calculated from the dot product between each direction vector. Note that -SC(ab) represents the dot product of vector a and vector b, and each angle of view can be calculated using the following formula: θ' αβ = cos －１ (-Sc([P Ｔ (N Ｖ ) N α ][P Ｔ (N Ｖ ) N β ]))...(19) θ' βγ = cos －１ (-Sc([P Ｔ (N Ｖ ) N β ][P Ｔ (N Ｖ ) N γ ]))...(20) θ' γα = cos －１ (-Sc([P Ｔ (N Ｖ ) N γ ][P Ｔ (N Ｖ ) N α ])) ... (21)

[0117] According to the theorem of circular angles, the angle between the external feature point α' and the external feature point β' is θ' αβThe set of points where X' is the terminal position is a point somewhere on the circumference of this circle. The same can be said for other sets of feature points. Therefore, three circles are created in total, and the location of terminal position X' in the horizontal projection plane of HMD 1B can be determined as the intersection of two of these circles (Figure 13).

[0118] To determine the specific coordinate value of X', select a horizontal projection plane, which passes through β and is Ｖ A plane perpendicular to β is selected. In other words, β' is equal to β. A two-dimensional coordinate system is defined on this plane. Hereafter, it is called coordinate system A1. The origin is β. A direction vector N from β to α' is Ｔ１１ is one of the coordinate axes. Direction vector N Ｔ１１ is given by the following formula: N Ｔ１１ = [P Ｔ (N Ｖ )(α-β)]...(22)

[0119] And the direction vector N Ｔ１１ , N Ｖ A direction vector N rotated 90 degrees around the axis Ｔ１２ Let N be the other coordinate axis. Ｔ１２ is given by the following formula: N Ｔ１２ = H(N Ｖ ) N Ｔ１１ H (N Ｖ ) ＊ ...(23)

[0120] Here, H(n) is a 90 degree rotation around the axis of the direction vector n (defined in equation (3)).

[0121] The numerical expression in the above coordinate system A1 is represented by a superscript (A1). The numerical expression of an arbitrary point D in the coordinate system A1 can be expressed in component notation as follows: D （Ａ１） = (-Sc(N Ｔ１１ P Ｔ (N Ｖ )(D-β)), -Sc(N Ｔ１２ P Ｔ (N Ｖ )(D-β)))...(24)

[0122] α' （Ａ１） , β' （Ａ１） , γ'（Ａ１） The coordinate value and the angle of view (θ' αβ , θ' βγ , θ' γα ) has been found, the coordinate value X' is calculated using the method for finding X' from the intersection of the circles described above. （Ａ１） The coordinate value X' thus obtained is （Ａ１） The component expression of X' is as follows: （Ａ１） = ((X' （Ａ１） ) １ , (X' （Ａ１） ) ２ ) ... (25)

[0123] Similarly, if the terminal position X'' of the HMD 1B in the vertical projection plane is found, the position of the vehicle-mounted terminal 1A of the HMD 1B in the terminal world coordinate system can be determined. Here, the horizontal direction vector N Ｈ It is necessary to determine the horizontal vector N Ｈ In this case, the vertically upward direction vector N Ｖ is a direction vector rotated 90 degrees around the axis of the direction vector from X' to β, and N Ｈ Define N Ｈ The numerical expression in each coordinate system is as follows: Ｈ （Ａ） = H([β （Ａ） -X' （Ａ） ]) Nv （Ａ） H ([β （Ａ） -X' （Ａ） ]) ＊ ... (26) N Ｈ （Ｂ） = H([P Ｔ (N Ｖ （Ｂ） ) N β （Ｂ） ]) Nv （Ｂ） H ([P Ｔ (N Ｖ （Ｂ） ) N β （Ｂ） ]) ＊ ...(27)

[0124] Here, [β' in formula (26) （Ａ） -X'（Ａ） ] is represented by the following formula, and X' （Ａ１） can be determined if [β （Ａ） -X' （Ａ） ]=-[(X' （Ａ１） ) １ N Ｔ１１ （Ａ） +(X' （Ａ１） ) ２ N Ｔ１２ （Ａ） ]・・・(28)

[0125] This N Ｈ By projecting each position onto the vertical projection plane using the above equation and performing the same processing as that performed within the horizontal projection plane, the position of the terminal position X'' within the vertical projection plane of the HMD 1B can be determined.

[0126] First, similarly to the case in the horizontal projection plane (FIG. 13), the angle of view (θ'') between the external feature points as seen from the position of the HMD 1B (terminal position X'') in the vertical projection plane is calculated. αβ , θ′′ βγ , θ′′ γα ) can be calculated using the following formula: θ'' αβ = cos －１ (-Sc([P Ｔ (N Ｈ ) N α ][P Ｔ (N Ｈ ) N β ]))...(29) θ'' βγ = cos －１ (-Sc([P Ｔ (N Ｈ ) N β ][P Ｔ (N Ｈ ) N γ ]))...(30) θ'' γα = cos －１ (-Sc([P Ｔ (N Ｈ ) N γ ][P Ｔ (N Ｈ ) N α ])) ... (31)

[0127] Then, similarly to the processing within the horizontal projection plane, the position of the terminal position X'' within the vertical projection plane of the HMD 1B can be determined as the intersection of the circles.

[0128] To find the specific coordinate value of X'', select one vertical projection plane. Here, the plane passing through β is N Ｈ Select a plane that is perpendicular to . In other words, β'' is equal to β. The vertical projection plane selected here is a plane obtained by rotating the vertical projection plane selected above by 90 degrees around the axis of the direction vector from X' to β. In other words, X' is located within this vertical projection plane.

[0129] Now, let us define a two-dimensional coordinate system on this plane. Hereafter, we will call it coordinate system A2. The origin is β. The direction vector N from β to α'' is Ｔ２１ is one of the coordinate axes. Direction vector N Ｔ２１ is given by the following formula: N Ｔ２１ = [P Ｔ (N Ｈ )(α-β)]...(32)

[0130] And the direction vector N Ｔ２１ , N Ｈ A direction vector N rotated 90 degrees around the axis Ｔ２２ Let N be the other coordinate axis. Ｔ２２ is given by the following formula: N Ｔ２２ = H(N Ｈ ) N Ｔ２１ H (N Ｈ ) ＊ ...(33)

[0131] The numerical expression in the above coordinate system A2 is expressed by a superscript (A2). The numerical expression of an arbitrary point D in the coordinate system A2 can be expressed in component notation as follows: D （Ａ２） = (-Sc(N Ｔ２１ P Ｔ (N Ｈ )(D-β)), -Sc(N Ｔ２２ P Ｔ (N Ｈ )(D-β)))...(34)

[0132] α'' （Ａ２） , β'' （Ａ２） , γ′′ （Ａ２） The coordinate value and the angle of view (θ'' αβ , θ′′ βγ , θ′′ γα) has been found, the coordinate value X'' can be calculated by the same method as in Figure 13 to find X'' from the intersection of the circle. （Ａ２） The coordinate value X'' obtained in this way is （Ａ２） The component expression of X'' is as follows: （Ａ２） = ((X'' （Ａ２） ) １ , (X'' （Ａ２） ) ２ ) ... (35)

[0133] From the above results, the numerical expression X in the vehicle-mounted terminal 1A of the terminal position X of the HMD 1B is （Ａ） Here, it should be noted that X' is a point included in both the horizontal and vertical projection planes, depending on how the horizontal and vertical projection planes are selected (Fig. 14). Therefore, the positional relationship between X' and X is N Ｖ If we pay attention to the alignment of the points in the direction, X is a point in the selected vertical projection plane. Therefore, the position of point X'' projected from X onto the vertical projection plane is equal to X. Therefore, X （Ａ） is calculated using the following formula: X （Ａ） =X'' （Ａ） = (X'' （Ａ２） ) １ N Ｔ２１ （Ａ） + (X'' （Ａ２） ) ２ N Ｔ２２ （Ａ） +β （Ａ） ...(36)

[0134] Here, we will explain points to note when estimating the terminal position X of the HMD 1B from the in-vehicle terminal 1A using the theorem of circular angles shown in FIG. 13 . As shown in FIG. 15 , there are cases where the three external field feature points α', β', and γ' and the terminal position X' in the horizontal projection plane of the HMD 1B are located on the same circumference. In this case, the terminal position X' of the HMD 1B cannot be determined using the process shown in FIG. 13 . In this case, another external field feature point that is not located on the same circumference is substituted (substitute external field feature point). The substitution with the substitute external field feature point is achieved by the processor 125A of the in-vehicle terminal 1A automatically setting a fourth point when the in-vehicle terminal 1A executes the process of FIG. 13 using the three external field feature points α', β', and γ' but is unable to estimate the terminal position X'. The same applies to processing in the vertical projection plane. To prepare for such cases, it is desirable to set the number of external field feature points whose positions are to be measured to four or more.

[0135] Below, we will determine the three-dimensional positional relationship between the user information terminal and the other person's information terminal, but in a normal meeting situation, the user information terminal and the other person's information terminal are located approximately on the same plane, so it is sufficient to only determine the positional relationship within the horizontal projection plane.

[0136] Through the processing up to this point, the terminal position X of the HMD 1B and the positions of each external feature point in the coordinate system of the vehicle-mounted terminal 1A are known. Now, if any two points are selected from the external feature points, for example, the external feature point β and the external feature point γ, a direction vector N from the terminal position X to the external feature point β is obtained. １ and a direction vector N from the terminal position X to the external feature point γ. ２ The numerical expression in the coordinate system of the vehicle-mounted terminal 1A is obtained (FIG. 16).

[0137] On the other hand, in the HMD 1B, the position and orientation sensor 135B of the HMD 1B detects the coordinate value of the HMD 1B in the coordinate system of the HMD 1B (i.e., the coordinate value of the terminal position X) and the direction vector N １ and N ２ We know the numerical representation of N １ and N ２ It is assumed that external feature points are selected in advance so that they do not have the same direction.

[0138] Now, the direction vector N in two different external worlds １ and N ２ Since the numerical expressions in the respective terminal world coordinate systems are obtained, the rotation Q is used to convert the numerical expression of the direction vector expressed in the coordinate system of the HMD 1B into the numerical expression of the direction vector expressed in the coordinate system of the vehicle-mounted terminal 1A by the method described in the method (1) for acquiring the positional relationship between the terminals. ＢＡ can be obtained (FIGS. 10 and 11).

[0139] Next, similarly to equation (16), the difference between the coordinate system origins is calculated using information on points whose numerical expressions in the coordinate systems of both terminals are known. ＢＡ is as follows: O ＢＡ =X （Ａ） -Q ＢＡ X （Ｂ） Q ＢＡ ＊ ...(37)

[0140] Here, X （Ａ） , X （Ｂ） are numerical representations of the terminal position X in the coordinate system of the vehicle-mounted terminal 1A and the coordinate system of the HMD 1B, respectively.

[0141] As with the method (1) for obtaining the positional relationship between terminals, the rotation Q ＢＡ and the difference between the origin position O ＢＡ By sharing this information between both terminals, each terminal can obtain the position coordinates of the other terminal in its own coordinate system and convert them into coordinates in its own coordinate system. In other words, both terminals can grasp the positional relationship between the terminals. Alternatively, one terminal may perform the coordinate conversion calculation and share the calculation results.

[0142] (Modification of Method (2) for Acquiring Positional Relationship Between Terminals) In the above, the projection plane is set based on the vertical direction that can be recognized by both terminals, but this is not limited to this as long as it is an external direction that can be recognized by both terminals. For example, the direction of north may be measured by the geomagnetic sensors 117A and 117B and used as the reference direction. Also, for example, the direction of a landmark that is sufficiently far from the terminals may be used as the reference direction. This reference direction vector is referred to as "N" in the above processing.Ｖ " and the same procedure can be performed.

[0143] (Summary of methods (1) and (2) for acquiring the positional relationship between terminals) Here we will summarize the methods for acquiring the positional relationship between two terminals. Hereinafter, the entity that processes data is a processor installed on one of the terminals or the server. The data to be processed is sent from each terminal to the terminal or server that will perform the processing, and the terminal or server that performed the data processing sends the processing results to the terminal or server that will use the processing results. Ultimately, each terminal uses the data processing results to recognize the relative position of the other terminal with respect to its own terminal, and uses this as support information for executing the rendezvous.

[0144] (1) The approximate location of the meeting is recognized by both terminals. This recognition method may involve, for example, sharing the name of the meeting place. The meeting place may be specified by the name of a street, the name of a nearby building or facility, or an address. Alternatively, location recognition information obtained by a common positioning system implemented on both terminals may be used. This common positioning system may be, for example, a positioning method that uses satellites such as GPS. Note that in the present invention, even if the specific positioning methods are different, those that use common location expressions such as latitude, longitude, and altitude are considered to be the same common positioning system.

[0145] (2) Both devices capture images of the outside world near the meeting place (first and second outside images). Because the same area of ​​the outside world needs to be captured, information about the object to be captured may be shared between both devices. The information about the object to be captured may include, for example, the name of the building or facility to be captured, the position of the object to be captured based on a common positioning system, or information about the location and direction of the image to be captured based on a common positioning system.

[0146] (3) Either one of the terminals or the server compares the images captured by both terminals and performs matching of external feature points. Distributed processing is also acceptable.

[0147] (4) In either the terminal or the server, for a pair of matching external feature points in the set of first external feature points and external feature points in the set of second external feature points, information on the relative positional relationship between the external feature points and the terminal in the positioning system of each terminal is obtained. Here, "information on the relative positional relationship between the external feature points and the terminal" refers to "information on the coordinate values ​​of the terminal position and the coordinate values ​​of the external feature points" or "information on the coordinate values ​​of the terminal position and the direction of the external feature points as seen from the terminal position." For at least one of the terminals, "information on the relative positional relationship between the external feature points and the terminal" needs to be "information on the coordinate values ​​of the terminal position and the coordinate values ​​of the external feature points." Here, "the positioning system of each terminal" refers to a positioning system by which each terminal recognizes its own position and orientation, and more specifically, refers to a measurement method and expression format for recognizing coordinate values ​​and direction vectors in a terminal world coordinate system defined by each terminal.

[0148] The direction of the external feature point as seen from the terminal position in the positioning system of each terminal described above can be calculated from the external world image captured by that terminal, as shown in Fig. 8. The calculation itself can be performed by other terminals or servers if the terminal position coordinates and image angle of view information are transmitted together with the external world image information.

[0149] In order to obtain the coordinate values ​​of the external feature points in the positioning system of each terminal described above, it is necessary to measure the distance from the terminal position to the external feature point in addition to information on the direction of the external feature point as seen from the terminal position. Distance measurement methods are broadly divided into methods using ranging sensors 113A, 113B, and 113C, and a moving stereo method using images captured at different shooting locations. While methods using ranging sensors 113A, 113B, and 113C are performed at the terminal, the moving stereo method may transmit image information from different shooting locations together with information on the shooting locations to a partner terminal (another person's information terminal as seen from the user information terminal, or the user information terminal as seen from the other person's information terminal) or a server, and the distance may be calculated at the partner terminal or server. Hereinafter, in distance measurement using the moving stereo method, it is assumed that the executing entity holds the information on the shooting location and image information required for measurement, or acquires that information from the shooting terminal. Furthermore, it is necessary to match external feature points in the images between the multiple images used in the moving stereo method, and this matching can be performed using the same method as the external feature point matching between the images of both terminals.

[0150] The distance between the device and the external world feature point may be measured before performing external world feature point matching. In this case, the device that performed the distance measurement transmits list information that pairs the position information of the external world feature in the external world image with the distance information to the device or server that uses the distance information. Here, the execution of distance measurement may be limited to the external world feature points extracted as external world feature points, or distance measurement to measurable external world points may be performed. In this method of measuring distance before external world feature point matching, there is a possibility that distance information is not available for the matched external world feature point. In this case, the matching may be treated as not being successful, or, as described below, distance measurement of the relevant external world feature point may be performed after external world feature point matching.

[0151] Measurement of the distance between the terminal and the external feature points may be performed after performing external feature point matching. In this case, if the terminal or server performing the distance measurement is not performing the matching process, the terminal or server that performed the matching process transmits information about the external feature points for which matching was achieved to the terminal or server performing the distance measurement. After the distance measurement is performed, the entity performing the distance measurement transmits the distance information to the terminal or server that requires it.

[0152] (5) Using the method (1) or (2) for acquiring the positional relationship between terminals, the terminal or server determines the "rotation and origin difference," which is the coordinate system transformation parameter between the world coordinate systems of both terminals. After the coordinate system transformation parameter is determined, the executing entity for calculating the coordinate system transformation parameter transmits the information to the terminal or server that uses the coordinate system transformation parameter.

[0153] Here, we summarize how to calculate coordinate system transformation parameters in methods (1) and (2) for acquiring the inter-device positional relationship, including both methods. First, a rotation parameter for aligning the orientations of the two device world coordinate systems (first device world coordinate system and second device world coordinate system) is calculated from the direction vector values ​​of two direction vectors defined in the external world in two device world coordinate systems. Next, the difference between the origins of the two device world coordinate systems is calculated from the coordinate values ​​of one position defined in the external world in the two device world coordinate systems. Here, the two direction vectors defined in the external world refer to two direction vectors selected from the direction vector from the device position to the external world feature point position or the direction vector from one external world feature point to another external world feature point. Furthermore, the one position defined in the external world refers to either one of the device positions or one external world feature point position.

[0154] (6) Each terminal transmits its own location information to the other terminal or server that uses this location information in advance or each time it is updated. This location information may be a numerical representation of the terminal's positioning system, or may be converted into a numerical representation of the other terminal's positioning system after the coordinate system transformation parameters are calculated. Alternatively, both the numerical representations of the positioning systems of both terminals may be transmitted. Each terminal or server executes a process to support rendezvous based on the terminal's location information. Examples of rendezvous support processes include superimposing the location of the other terminal on an external image, or using the information to control the driving of a vehicle such as a car to guide the other terminal to its location.

[0155] (Modification (1) common to methods (1) and (2) for acquiring positional relationship between terminals) In the method of the present invention, a direction vector to an external feature point is acquired from an external image captured by the terminal, so it is preferable that distortion of the optical system is corrected and distortion of the captured image is minimized. Therefore, if the outer cameras 111A, 111B, and 111C of the terminal have a low-distortion shooting mode, the external image for acquiring the relative position may be automatically captured in the low-distortion shooting mode. Figure 17 shows an example in which a message for switching to low-distortion shooting mode is superimposed on the captured image preview screen. Switching to low-distortion shooting mode can improve the accuracy of estimating the relative position.

[0156] (Variation (2) common to methods (1) and (2) for acquiring positional relationship between terminals) In the method of the present invention, it is essential that both terminals capture images of the same area of ​​the outside world, so the image used to acquire the relative position may be wide-angle image. Therefore, if the outer cameras 111A, 111B, and 111C of the terminals have a wide-angle mode, wide-angle image may be automatically captured when capturing image for acquiring the relative position. Figure 18 shows an example in which a message for switching to wide-angle capture mode is superimposed on the captured image preview screen. Switching to wide-angle capture mode increases the certainty of capturing images of the same area of ​​the outside world, reducing the risk of taking additional images.

[0157] (Modification (3) common to methods (1) and (2) for acquiring the positional relationship between terminals) As with modification (2) described above, a shooting guide may be displayed as a method for increasing the certainty that both terminals will shoot the same area of ​​the outside world. The shooting guide here is to display the image shot by the other terminal on the shot image preview screen of the terminal itself when the other terminal shoots an image of the outside world first. An example of such a display is shown in FIG. 19. By displaying the image shot by the other terminal and providing the user with guidance on the shooting direction, the certainty that the same area of ​​the outside world will be shot increases, and the risk of taking additional images can be reduced.

[0158] (Modification (4) common to methods (1) and (2) for acquiring positional relationship between terminals) In addition to the above-described modification (3), another shooting guide may be displayed as a method for increasing the certainty that the same area of ​​the outside world is photographed on both terminals. The shooting guide here is to display an AR object or the like indicating the target shooting point on the shooting video preview screen of the terminal itself, based on target shooting point information based on a common positioning system such as GPS that allows both terminals to share position recognition. An example of such a display is shown in FIG. 20. By displaying an AR object or the like indicating the target shooting point on the shooting video preview screen of the terminal itself, the certainty that the same area of ​​the outside world is photographed is increased, and the risk of taking additional photographs can be reduced.

[0159] (Modification (5) common to methods (1) and (2) for acquiring positional relationship between terminals) In the above process, three matched external feature points are used (one or two external feature points are used when a specific direction in real space is used) to calculate the rotation and origin difference, which are the transformation parameters of the coordinate systems of both terminals. However, more external feature points may be used. That is, multiple coordinate system transformation parameters may be obtained from multiple sets of three external feature points, and their average value may be used. When a specific direction in real space is used, the number of external feature points is determined to be the number required for that case. In this case, when looking at the distribution of the coordinate system transformation parameters, any data that deviates significantly from the average value may be considered anomalous data and excluded from the data for calculating the average value, and the calculation may be repeated. Well-known methods such as the Smirnoff-Grubbs test may be used to determine anomalous data. For example, if the standard deviation of the distribution is σ, data that deviates by 3σ or more from the average value may be determined to be anomalous data.

[0160] The process of calculating the average value may be performed on the coordinate values ​​in the terminal's own world coordinate system of the location of the other terminal estimated using the coordinate system transformation parameters. In this case, when examining the distribution of estimated coordinate values, any values ​​that deviate significantly from the average value may be regarded as abnormal data and excluded from the data for calculating the average value, and the calculation may be retried. The coordinate system transformation parameters corresponding to these abnormal estimated coordinate values ​​are also recognized as abnormal data.

[0161] Among the three external field feature points corresponding to coordinate system transformation parameters determined as abnormal data using the above-described method, there is a possibility that at least one is a false external field feature point that has been accidentally matched between two images. Therefore, external field feature points belonging to a triplet of external field feature points corresponding to abnormal data equal to or greater than the set number of pairs may be determined as false external field feature points, and the matching may not be established. In other words, this is additional screening of external field feature point matching based on the calculation results of coordinate system transformation parameters.

[0162] After the above-described additional screening of external boundary feature points is performed, a plurality of coordinate system transformation parameters may be newly calculated from a plurality of triplet sets of external boundary feature points, and the process of calculating the average value thereof may be repeated.

[0163] By performing the processing of this modified example, it is possible to improve the accuracy of estimating the relative position.

[0164] (Variation (6) Common to Methods (1) and (2) for Acquiring Positional Relationship Between Terminals) If the number of matched external world feature points is less than a set number, or if the variation in the transformation parameters of the coordinate systems of both terminals calculated using the matched external world feature points is greater than a set amount, it is possible that the locations captured by both terminals are different, or that the captured ranges of the images have little overlap. In this case, if additional capture is possible, a message urging the users of both terminals to take additional capture may be displayed by visual display or audio notification. This increases the likelihood that the same area of ​​the external world will be captured if additional capture is performed ( FIG. 21A ). Furthermore, if it is determined that a terminal has moved away from the capture location, the terminal may stop issuing the above-mentioned message and notify the other terminal that it is unable to take additional capture, prompting the user of the other terminal to take additional capture ( FIG. 21B ). This improves the effectiveness of the message.

[0165] Furthermore, when the moving terminal (vehicle-mounted terminal 1A) approaches the meeting place, only the vehicle-mounted terminal 1A may additionally capture an external image of the vicinity of the meeting place. This changes the image on the vehicle-mounted terminal 1A side to an image captured from a location closer to the location captured by the HMD 1B. This increases the reliability of matching of external world feature points, increases the number of matching external world feature points, and improves the accuracy of estimating the relative positions of the two terminals.

[0166] (Information Display) FIG. 22 is a diagram showing an example of information display on the user information terminal (vehicle-mounted terminal 1A) based on terminal position estimation for meeting support.

[0167] 22 , an outside image 1200 captured by the outer camera 111A is displayed on the display 119A of the in-vehicle terminal 1A. Furthermore, virtual objects 1210 and 1211 indicating the terminal position X of the HMD 1B calculated by the processor 125A of the in-vehicle terminal 1A are displayed superimposed on the outside image 1200 of the display 119A. The terminal position X corresponds to the estimated position of the passenger.

[0168] FIG. 23 is a diagram showing an example of information display on the other person's information terminal (HMD 1B) based on terminal position estimation for meeting support.

[0169] 23 , an outside world image 1300 captured by the outer camera 111B is displayed on the display 119B of the HMD 1B. Furthermore, the HMD 1B receives information indicating the relative positional relationship between the in-vehicle terminal 1A and the terminal position X of the HMD 1B, calculated by the processor 125A of the in-vehicle terminal 1A. The processor 125B of the HMD 1B then calculates the terminal position of the in-vehicle terminal 1A relative to the HMD 1B using the information indicating the relative positional relationship, and displays virtual objects 1310 and 1311 indicating the position of the in-vehicle terminal 1A superimposed on the outside world image 1300 on the display 119B. The objects 1310 and 1311 are virtual objects indicating the estimated position of a taxi.

[0170] This embodiment makes it possible to provide highly accurate rendezvous guidance.

[0171] The information display in the automobile using the in-vehicle terminal 1A may be a HUD (Head Up Display).

[0172] If the estimated location of the other party's terminal is not included in the range shown on the display, the direction to be displayed may be guided by an arrow display, audio, etc. The direction of the meeting party may be presented by sound image localization using 3D audio.

[0173] Furthermore, the information on the location of the other party's terminal may be used not only to confirm the other party but also to control the speed of the car so that the car stops quickly and without abrupt stops in front of the other party. The speed control may be used to recommend a speed to the driver, or for speed control in a driving support system or an automatic driving system.

[0174] 24 is a flowchart showing the processing flow of the meeting support system 100 according to this embodiment. Although the server 2 is not explicitly shown, some of the processing may be performed by the server 2 to distribute the processing load.

[0175] First, the in-vehicle terminal 1A and the HMD 1B accept the setting of a meeting place by the user and start processing. A specific example of use is when a user of the HMD 1B requests a taxi to pick them up and sets the meeting place on the in-vehicle terminal 1A installed in the requested taxi. After setting the meeting place, the in-vehicle terminal 1A and the HMD 1B, for example, capture external world images near the meeting place using the outer cameras 111A and 111B, acquire terminal position coordinates at the time of capture in the respective terminal world coordinate systems, and measure the distance from the terminal position to an external world feature point (S01A, S01B). The distance measurement may be performed using distance measurement sensors 113A and 113B such as LiDAR or TOF, or a moving stereo method using multiple images captured at different positions. Note that if the method (2) for acquiring the positional relationship between terminals is used, distance measurement can be omitted in one of the terminals, so distance measurement is not performed in the terminal that omits distance measurement. Alternatively, distance measurement at this stage may not be necessary in the terminal that measures distances to external feature points after feature point matching.

[0176] Next, the in-vehicle terminal 1A and the HMD 1B share the captured external world images via the network communication devices 120A and 120B (S02A, S02B). Hereinafter, the transmission and sharing of information will be assumed to be performed via the network communication devices 120A, 120B, and 220, and an explicit description thereof will be omitted. Here, the external world image data may be aggregated only in the terminal that performs external world feature point matching or in the server 2.

[0177] Next, the processors 125A and 125B of the in-vehicle terminal 1A and the HMD 1B compare the images captured by both terminals, match the external feature points, and share the matching results (S03A, S03B). Here, the matching process may be executed by the processor of either terminal, or by the processor 225 of the server 2. Hereinafter, in text expressions in which the in-vehicle terminal 1A, the HMD 1B, or the server 2 is the processing subject, it is assumed that the processing subjects at the hardware component level shown in Figures 2 and 3 are the processors 125A, 125B, and 225.

[0178] Here, the vehicle-mounted terminal 1A and the HMD 1B determine whether the number of matching external feature points is equal to or greater than a predetermined number (S04A, S04B).

[0179] If the determinations in S04A and S04B are negative (S04A, S04B: No), the in-vehicle terminal 1A and the HMD 1B return to steps S01A and S01B. If one of the terminals, for example, the HMD 1B, has moved from the shooting point to a distant location and is unable to take additional images, the HMD 1B skips step S01B and repeats the process from S02B using the previous image.

[0180] If the determinations in S04A and S04B are positive (S04A and S04B: Yes), the in-vehicle terminal 1A and the HMD 1B perform distance measurement for the matched external field feature points (S05A and S05B). If distance measurement has been performed in the steps of S01A and S01B, distance measurement at this step may be omitted. Furthermore, distance measurement is not performed in terminals that omit distance measurement.

[0181] In the distance measurement in steps S05A and S05B, if the terminal has not moved from the shooting point of the video used for matching, distance measurement is possible using distance measurement sensors 113A and 113B such as LiDAR or TOF. If the terminal moves and distance measurement of the external world feature point by distance measurement sensors 113A and 113B becomes impossible, the in-vehicle terminal 1A and HMD 1B capture multiple images in advance with the outer cameras 111A and 111B by shifting the shooting point within a range where the same external world area can be captured, and measure the distance from the shooting point to the external world feature point by the moving stereo method.

[0182] Next, the in-vehicle terminal 1A and the HMD 1B calculate the coordinates of the external world feature points in their respective terminal world coordinate systems using the results of the distance measurements from the terminal position to the external world feature points performed up to this stage, and share the terminal position coordinates and the external world feature point coordinates (S06A, S06B). Note that the calculation of the external world feature point coordinates can be performed after the distance measurement, so it may be performed before this stage. Furthermore, the in-vehicle terminal 1A and the HMD 1B may transmit the results of the distance measurement to another terminal or the server 2 to have the external world feature point coordinates calculated, and share the results.

[0183] Next, the in-vehicle terminal 1A and the HMD 1B use the terminal position coordinates and the external world feature point coordinates to calculate and share transformation parameters between the world coordinate systems of both terminals (S07A, S07B). Note that the in-vehicle terminal 1A and the HMD 1B may transmit information on the terminal position coordinates and the external world feature point coordinates to a terminal or server 2 that performs calculation processing of the terminal world coordinate system transformation parameters, have the transformation parameters between the world coordinate systems calculated, and share the results. The process of having the other party's terminal or server 2 perform the calculation may be performed consecutively from the stages of steps S06A and S06B.

[0184] Next, the in-vehicle terminal 1A and the HMD 1B transmit their own current position information to each other terminal, and use the conversion parameters between the world coordinate systems to grasp the position of the other terminal in the world coordinate system (S08A, S08B). This calculation of the positions of each terminal may also be performed in one of the terminals or in the server 2.

[0185] At this point, the vehicle-mounted terminal 1A and the HMD 1B determine whether to update the external world image (S09A, S09B). This determination is made based on whether the variation in the estimation result of the partner terminal position exceeds a certain limit and on how close the moving terminal (vehicle-mounted terminal 1A) has come to the meeting place.

[0186] The variance of the estimation results is determined by, for example, updating the external world image when the standard deviation of the estimated position distribution is equal to or greater than a predetermined value. The estimated positions are calculated as three-dimensional or two-dimensional values, and the standard deviation mentioned above refers to the standard deviation of the distribution of distances of each estimated position from the average position.

[0187] The degree of approach of the vehicle-mounted terminal 1A to the meeting place is determined by the distance between the terminal position of the vehicle-mounted terminal 1A and the estimated terminal position of the HMD 1B. The image is updated when a predetermined threshold value for the estimated inter-terminal distance is reached. Multiple threshold values ​​may be defined. A standard deviation of the target estimated position is determined for each threshold value, and it is determined that the image should be updated if the estimated position is not equal to or less than that standard deviation.

[0188] If the determinations in S09A and S09B are positive (S09A, S09B: Yes), the in-vehicle terminal 1A and the HMD 1B return to the processing of steps S01A and S01B. However, once the determinations in S09A and S09B become negative (S09A, S09B: No) and the output of meeting support information (S10A, S10B) has started, the in-vehicle terminal 1A and the HMD 1B continue the processing of steps S08A, S08B, S10A, and S10B in parallel using the existing coordinate transformation parameters until the external world image is updated and new coordinate transformation parameters are obtained according to the flow after returning to the processing of steps S01A and S01B.

[0189] If the determinations in S09A and S09B are negative (S09A, S09B: No), the in-vehicle terminal 1A and the HMD 1B output meeting support information (S10A, S10B). Note that if the processing of steps S08A, S08B, S10A, and S10B has been continued using existing coordinate transformation parameters, the in-vehicle terminal 1A and the HMD 1B change to newly obtained coordinate transformation parameters and continue the processing of steps S08A, S08B, S10A, and S10B until the termination condition (S11A, S11B) is satisfied.

[0190] In steps S10A and S10B, the in-vehicle terminal 1A and the HMD 1B each output information for meeting support based on the estimated position of the other terminal. Note that in the in-vehicle terminal 1A, the information output for meeting support may be for automatic driving control. In addition, in the HMD 1B, if the user simply waits at the meeting place, there is no need to use information for meeting support, such as displaying the estimated position of the other terminal.

[0191] Next, if the termination condition is satisfied (S11A, S11B: Yes), the in-vehicle terminal 1A and the HMD 1B terminate the process, and if the termination condition is not satisfied (S11A, S11B: No), the process returns to steps S01A and S01B. The termination condition for the in-vehicle terminal 1A may be determined to be satisfied, for example, when the in-vehicle terminal 1A arrives at the meeting place, picks up a passenger, and turns on the taxi meter. The termination condition for the HMD 1B may be determined to be satisfied when the distance between the terminal positions of the in-vehicle terminal 1A and the HMD 1B is less than a distance threshold, for example, 1 meter.

[0192] In this embodiment, both devices capture images of the external world around the meeting place at the time of the meeting, and the location of the other device in its own device's world coordinate system is estimated by comparing the numerical representations of external feature points in the images in each device's world coordinate system. This makes it possible to easily and accurately estimate the location of the other device without being affected by position measurement errors in a general-purpose common positioning system, such as reflection of satellite radio waves by buildings, and without the need to prepare a huge amount of external world images of the area around potential meeting places in advance. Furthermore, compared to using library images, this invention uses images from the time of the meeting, which has the advantage of not being affected by changes in the image that may occur between the time the library images were captured and the time of use.

[0193] <Second Embodiment> In a second embodiment, in order to perform more precise speed control of the vehicle, some or all of the external field feature points may be set on the body of the vehicle on which the in-vehicle terminal 1A is mounted. Fig. 25 is a diagram showing an example of setting the external field feature points in the second embodiment. In this case, the image of the vehicle used for matching the external field feature points is one that has been captured in advance. If it is possible to identify which part of the vehicle body is involved, the coordinate values ​​in the terminal local coordinate system can be determined, and by referring to the time of capturing the image on the HMD 1B, the coordinate values ​​in the terminal world coordinate system can be calculated.

[0194] 25, in a situation where a car is approaching the vicinity of the meeting point, the external world feature points set on the car body will be closer to the HMD 1B than the external world feature points set on a building, etc., and furthermore, for the in-vehicle terminal 1A, these are external world feature points whose position coordinates are known accurately, so the accuracy and precision of position estimation will be improved more by setting external world feature points on the car body than by setting external world feature points on a building, etc. In this case, in order to confirm that the car is the one to be met, an identification number or the like displayed on the car body that can be seen in the video may be recognized.

[0195] <Third embodiment> In a third embodiment, the information display (FIG. 22) of the in-vehicle terminal 1A may also be displayed on the HMD 1B, and the passenger may check whether the passenger has been correctly identified by estimating the position of the destination point. If the passenger has been incorrectly identified, the HMD 1B may transmit a message to that effect to the in-vehicle terminal 1A. The passenger may manually input the determination as to whether the passenger has been correctly identified into the HMD 1B, or the processor 125B of the HMD 1B may automatically determine whether the passenger has been correctly identified.

[0196] Furthermore, if the passenger is incorrectly captured in the image sent from the vehicle-mounted terminal 1A, the true passenger position can be indicated to the vehicle-mounted terminal 1A based on the position in the image, thereby correcting the passenger's identification. If the passenger is not captured in the image sent from the vehicle-mounted terminal 1A, the HMD 1B may notify the vehicle-mounted terminal 1A of the direction in which the true passenger position deviates from the image. Furthermore, if the estimation of the passenger position is incorrect, both terminals may re-capture the external world image around the target point, and the relative position may be re-estimated.

[0197] After the passenger to be met is confirmed on the video, the speed of the vehicle may be controlled so as to stop the vehicle at a road position nearest to the passenger.

[0198] According to this embodiment, the destination can be confirmed by other passengers using the information terminal. In addition, by diverting the technology to speed control, it is expected that the accuracy of stopping an autonomous taxi vehicle in front of a passenger can be improved.

[0199] Fourth Embodiment The fourth embodiment is a modified example in which the photographing point on the HMD 1B side, which is the other person's information terminal, is different from the meeting place (FIG. 26).

[0200] In cases where the meeting place is a narrow road off a main road and video captured from the meeting place is not suitable for meeting support, for example, video captured from a position other than the meeting place may be sent to the vehicle-mounted terminal 1A as a comparison video. In this case, since the HMD 1B can grasp its own position in the world coordinate system using the position and orientation sensor 135B, it transmits the coordinate values ​​of the meeting place (target point) in addition to the coordinate values ​​of the shooting point to the vehicle-mounted terminal 1A. The vehicle-mounted terminal 1A can grasp the meeting place by converting the coordinate values ​​of the meeting place in the terminal world coordinate system of HMD 1B into coordinate values ​​in the terminal world coordinate system of the vehicle-mounted terminal 1A using the transformation parameters between coordinate systems obtained by the method of embodiment 1.

[0201] Furthermore, the user of the HMD 1B may be waiting for a taxi inside a building. In this case, the HMD 1B may transmit the coordinate values ​​of a point in front of the building as the meeting place to the vehicle-mounted terminal 1A, and may transmit an image of the building where the user of the HMD 1B is waiting to the vehicle-mounted terminal 1A for confirmation of the meeting place.

[0202] According to this embodiment, video is captured from a shooting position that is different from the meeting place, data necessary for creating external world feature point information is measured, and then the vehicle can move to the original meeting place. Therefore, even if the meeting place is difficult to find because it faces a narrow alley, by setting up a shooting position on a main street, it becomes easier to support a taxi in approaching the meeting place. It is also possible to wait for a taxi at a location away from the meeting place. Note that the external world feature point information in this case is information that directly indicates the relative positional relationship between the external world feature points and the video shooting position, but it can also be said to be information that indirectly indicates the relative positional relationship between the external world feature points and the information terminal.

[0203] Fifth Embodiment In a fifth embodiment, images of external feature points captured by passengers may be provided to a taxi company to be included in a video library. In this case, passengers may be given an incentive, such as a discounted fare. Because the state of the external world may change over time, it is desirable to keep updating the video library with as much new video as possible. This allows the vehicle-mounted terminal 1A to perform location estimation based on the latest situation when using library video, thereby improving the accuracy of location estimation.

[0204] Sixth Embodiment This embodiment is a case where two users are both moving around to find each other. For example, a meeting is planned in a place where GPS signals cannot reach, such as a shopping mall, with only a rough location, such as a certain floor, being decided. In this embodiment, the process is described in the form of a meeting between users of a first information terminal and a second information terminal, but in this embodiment, it is assumed that the first information terminal and the second information terminal may be mobile terminals such as HMDs or smartphones.

[0205] FIG. 27 is a diagram illustrating the movement of the information terminal in the sixth embodiment.

[0206] A use case of this embodiment will be described with reference to Fig. 27. Fig. 27 shows the trajectories of a user of a first information terminal and a user of a second information terminal who move around an area of ​​a meeting place (corresponding to the "meeting place area") in a building in search of the other person. In the figure, the trajectories approach each other at a location indicated as the "trajectory approaching portion." The meeting support method of this embodiment uses external feature points (referred to as "external feature points") near this location to grasp the relative positional relationship between the two terminals.

[0207] FIG. 28 is a diagram for explaining the video matching process in the sixth embodiment, and shows the concept of the process for grasping the current relative positional relationship between both terminals.

[0208] First, both devices acquire images of the surrounding environment while moving, and simultaneously record changes in their position and orientation. This information on the images of the surrounding environment, the device's position, and the device's orientation is then shared between the two devices. Alternatively, data can be aggregated on one device or on a server to perform relative position calculations, and the calculation results can be notified to the device that is not performing the calculations.

[0209] To calculate the relative position, first, past images of the outside world taken by both terminals are searched for images of the same location. This location is the trajectory approach section in Figure 27. From the images of the outside world taken by both terminals at the same location in this trajectory approach section, coordinate transformation parameters between the coordinate systems of both terminals are calculated using the method described above. Once the coordinate transformation parameters are known, the current coordinate values ​​of the other terminal's position can be converted to the coordinate values ​​of the own terminal, thereby determining the positional relationship between the own terminal and the other terminal.

[0210] Here, to determine the relationship between the coordinate systems of the two devices, the coordinates of external feature points in the external world images are used, so it is necessary to measure the distance from the device to each external feature point on at least one device. This measurement can be performed using a moving stereo technique using multiple images taken from different terminal positions near the trajectory approach point. By using this method, it becomes possible to select feature points that appear in both images captured by both devices later and measure only the distance to those feature points from the previous and next images, without having to determine the external feature points to be used in advance. Point cloud data can also be obtained by measuring the distance to each point in the image using LiDAR or the like.

[0211] Here, the trajectory approach section has been described as a point that both terminals have passed through in the past, but if the video search speed is fast enough, a terminal that arrives at the trajectory approach section later can search the past video of the other terminal on the spot and recognize that the location at the trajectory approach section is the trajectory approach section.

[0212] In the example of FIG. 28, the number of matching external feature points is checked between frame tAm (m=0, 1, 2, . . . ) constituting the external image captured by the first information terminal and frame tBm (m=0, 1, 2, . . . ) constituting the external image captured by the second information terminal. １Once the frame is obtained, the number of matching external feature points between the frame and the already obtained frame tBm (m=0, 1, ..., m1) is checked. １ Once a frame tAm (m=0, 1, ..., m1) is obtained, the number of matching external feature points between the frame tAm and the frame tAm (m=0, 1, ..., m1) already obtained at the time the frame is obtained is checked. Note that, although the shooting time is used as the video index in FIG. 28, this is not limiting. Furthermore, the video shooting times of the first and second information terminals do not need to be the same.

[0213] During this procedure, a pair of frames (hereinafter sometimes referred to as a matching pair) that has a predetermined number or more of matching external field feature points is determined to be an image pair of the trajectory approach portion. In the example of FIG. 28 , a pair of frames tA1 and tB2 is selected as the trajectory approach portion image pair. That is, the position of the first information terminal when capturing the first external field image, which is one of the trajectory approach portion image pair, and the position of the second information terminal when capturing the second external field image, which is the other of the trajectory approach portion image pair, are determined to be the position of the trajectory approach portion. This method uses the first matching pair found for subsequent calculation of coordinate transformation parameters, but there may be multiple matching pairs. After several matching pairs are found, the matching pair with the largest number of matching external field feature points may be used for subsequent calculation of coordinate transformation parameters. The greater the number of matching external field feature points, the more likely it is that the accuracy of the coordinate transformation parameters will be improved. Note that, as a method for matching external field feature points, the method described in the first embodiment, for example, is used. When referring to external feature points in an image, the external feature points in the first external image are referred to as first external feature points, and the external feature points in the second external image are referred to as second external feature points.

[0214] In the above process, it is important to continuously measure changes in the terminal's position and orientation while capturing surrounding images. Here, we will show an example of measuring changes in the terminal's position and orientation using an acceleration sensor and a gyro sensor as position and orientation sensors.

[0215] FIG. 29 is a diagram illustrating the relationship between the terminal local coordinate system and the terminal world coordinate system.

[0216] After entering the waiting area, the position and orientation of each terminal are measured in their respective terminal positioning systems. Here, the position and orientation of the terminal with respect to the terminal world coordinate system can be expressed by a relational expression for coordinate transformation between the terminal local coordinate system and the terminal world coordinate system (see FIG. 29, Equation (11)).

[0217] The coordinate value O of the origin of this terminal's local coordinate system in the terminal's world coordinate system LW and the rotation Q from the reference point in the terminal's local coordinate system in the terminal's world coordinate system LW are the position and orientation of the terminal in the terminal world coordinate system. At the reference point in time, the orientation of the coordinate axes of the terminal local coordinate system is assumed to match the orientation of the coordinate axes of the terminal world coordinate system. Examples of reference points in time include the time when the terminal world coordinate system is initially set when the terminal is turned on, and the time when the terminal world coordinate system is reset. Each terminal measures its own position and orientation using its own terminal positioning method. This positioning is called "autonomous positioning." Here, an example of "autonomous positioning" using acceleration sensors 115A, 115B and gyro sensors 116A, 116B will be described as an example. The content is common to the first information terminal and the second information terminal.

[0218] First, the time t ０ The initial position at O LW０ , the initial orientation is Q LW０ After the initial state, the state is updated at a constant minute time interval Δt. First, the recurrence formula for the change in orientation is shown. Let ω be the numerical expression in the terminal local coordinate system of the angular velocity vector that is the output of the gyro sensors 116A and 116B at time t. Ｌ At this time, the orientation of the terminal at time Δt is ω Ｌ The angle |ω with the direction of (t) as the axis Ｌ (t) | Rotate by Δt. Let the quaternion that represents this rotation be Δq Ｌ Here, |q| is the norm of the quaternion q. This Δq Ｌ is expressed as follows: Ｌ = cos(|ω Ｌ (t) |Δt / 2) +SIN(|ω Ｌ (t) | Δt / 2) [ω Ｌ(t)] =1+(|ω Ｌ (t) | Δt / 2) [ω Ｌ (t)]+O((Δt) ２ ) = 1 + ω Ｌ (t)Δt / 2+O((Δt) ２ ) ... (38)

[0219] Here, O((Δt) ２ ) is (Δt) ２ This means that the quantity is on the order of Δq. Ｌ The numerical expression in the terminal world coordinate system is Δq W Then, the rotation Q LW The recurrence formula for is as follows:

[0220] Q LW (t + Δt) = Δq W Q LW (t) = Q LW (t) Δq Ｌ Q LW (t) ＊ Q LW (t) = Q LW (t) + Q LW (t)ω Ｌ (t)Δt / 2+O((Δt) ２ ) ... (39)

[0221] Rotation Q LW Since Q must be normalized, in actual processing, the calculation result of the right side of equation (39) may be normalized and used as the value at the next time (t+Δt) to prevent error propagation. Specifically, the calculation is as follows: Q LW (t + Δt) = [Q LW (t) + Q LW (t)ω Ｌ (t)Δt / 2]...(40)

[0222] Next, the recurrence formula for the position change is the acceleration a Ｌ This acceleration is also expressed in the terminal local coordinate system, so the expression in the terminal world coordinate system is a Ｗ When converted to (t), the following equation is obtained: Ｗ (t) = Q LW (t) a Ｌ (t)Q LW(t) ＊ ...(41)

[0223] Here, the outputs of the acceleration sensors 115A and 115B include the gravitational acceleration g in addition to the acceleration due to the change in the position of the terminal. Ｗ This gravitational acceleration g Ｗ can be obtained from the outputs of the acceleration sensors 115A and 115B when the terminal is stationary. The velocity of the terminal in the terminal world coordinate system is V Ｗ (t), the gravitational acceleration g Ｗ By correcting the component of V Ｗ The recurrence formula for (t) can be written as follows. The initial value of the terminal velocity is set to 0 when the terminal is stationary. V Ｗ (t+Δt)=V Ｗ (t) + (a Ｗ (t)-g Ｗ ) Δt (42)

[0224] V calculated in this way Ｗ Using (t), O LW The recurrence formula for (t) can be written as follows: LW (t + Δt) = O LW (t) + V Ｗ (t)Δt...(43)

[0225] In this way, the position and orientation of the terminal can be determined by positioning using acceleration sensors 115A and 115B and gyro sensors 116A and 116B. Note that in equations (42) and (43), too, the calculation results on the right-hand sides may be normalized and substituted for the variables on the left-hand sides in order to prevent propagation of errors.

[0226] To calculate the positional relationship between devices, the coordinate values ​​of each device in the device's world coordinate system are required. L0 When this is set, the value R in the terminal world coordinate system Ｗ0 is expressed as follows: R Ｗ0 =Q LW (t)R L0 Q LW (t) ＊ +O LW (t) ... (44) Unless there are special circumstances, R L0Setting it to 0 will save you the trouble of doing the calculations.

[0227] (Video Matching Method) Videos captured by a first information terminal and a second information terminal are matched based on the number of external world feature points that match between the videos. Specifically, each time a new video is captured while moving, the external world feature points are matched with video previously captured by the other terminal. Video pairs that have already undergone matching processing are excluded. Then, when a video pair that exceeds a predetermined number of matches is found, that video pair is used to find coordinate transformation parameters between the terminals using the method of the first embodiment.

[0228] Once the coordinate transformation parameters between the terminals are calculated, the movement route of the other terminal is known, so while moving towards the other terminal, additional images can be taken at the same locations as those taken by the other terminal, and new coordinate transformation parameters can be calculated to improve the accuracy of the coordinate transformation parameters. Specifically, the coordinate transformation parameters {Q ＢＡ , O ＢＡ} are obtained, so that every time new coordinate transformation parameters are obtained, the average value of the previously obtained coordinate transformation parameters is used.

[0229] The video matching and external feature point matching processes may be performed on both devices, on one device, or on the server. Each device transmits the video data it has captured to the device performing the matching process. At that time, the device transmits the coordinate values ​​of the video captured at the time of capture along with the external feature point information.

[0230] (Guidance Display) Fig. 30 is a diagram illustrating an example of guidance display in the sixth embodiment. Fig. 31 is a diagram illustrating another example of guidance display in the sixth embodiment.

[0231] After the location of the other terminal is determined, a display is made to guide the user to the estimated location of the other terminal. Since the movement routes of both terminals from the trajectory approach point are known, a guided route from the user's terminal to the other terminal along those movement routes can be displayed. An example of this guided route superimposed on the external world image as a virtual object is shown in Figure 30. This guided route follows the route taken by both terminals, so it is a route that the user can take.

[0232] In the present invention, when the HMD is a see-through type, the "external world image" is interpreted as the "visual image of the external world" that is visually recognized by the user.

[0233] Another example of guidance display is shown in Figure 31, which shows the direction and location of the other party's terminal. If the structure of the meeting area is known, this display method is simple because if the location of the other party's terminal is known, the shortest route may be found regardless of the route traveled up to that point.

[0234] (Flow Diagram) FIG. 32 is a flowchart showing the flow of processing in the sixth embodiment.

[0235] This flow chart is basically the same as the flowchart of the first embodiment (FIG. 24). The first information terminal corresponds to the user information terminal of the first embodiment, and the second information terminal corresponds to the other person's information terminal of the first embodiment. The correspondence may be interpreted as being reversed. Although the server 2 is not explicitly shown in FIG. 32, some of the processing may be performed by the server 2 to distribute the processing load.

[0236] The difference between the flow of this embodiment and that of the first embodiment is that in the first embodiment, matching of external world feature points is performed between external world images near the meeting place, whereas in this embodiment, matching of external world feature points is performed between external world images of multiple locations captured by both terminals while they are moving.

[0237] Specifically, steps S100A, S100B, S101A, and S101B are steps that differ from Fig. 24. The other steps are the same as those in Fig. 24, so the same numbers are used and the description will be omitted.

[0238] In steps 100A and 100B, matching of external feature points is performed between the video captured by the own terminal and the video of the moving route captured by the other terminal.

[0239] In steps 101A and 101B, it is determined whether or not there are a predetermined number of pairs for which matching has been achieved among the video pairs for which matching processing has been performed.

[0240] If the determination in S101A and S101B is negative (S101A, S101B: No), the first information terminal and the second information terminal return to steps S01A and S01B, and capture an external image of the next location.

[0241] If the determinations in S101A and S101B are positive (S101A and S101B: Yes), the first information terminal and the second information terminal perform distance measurement for matching external feature points in video pairs for which a predetermined number of matches or more have been established (S05A and S05B). If distance measurement has been performed in steps S01A and S01B, distance measurement at this step may be omitted. Furthermore, distance measurement is not performed in terminals that omit distance measurement.

[0242] In the distance measurement in steps S05A and S05B, if the terminal has not moved from the shooting point of the video used for matching, distance measurement is possible using ranging sensors 113A and 113B such as LiDAR or TOF. If the terminal moves and distance measurement of the external world feature point using ranging sensors 113A and 113B becomes impossible, the first information terminal and the second information terminal capture multiple images in advance with outer cameras 111A and 111B by shifting the shooting point within a range where the same external world area can be captured, and measure the distance from the shooting point to the external world feature point using a moving stereo method.

[0243] (Effects of the Present Embodiment) In the present embodiment, both devices capture images of the external world within the meeting area to be used at the time of the meeting, and compare information on external feature points in the images, expressed numerically in the respective device world coordinate systems, to estimate the location of the other device in its own device world coordinate system. This enables the location of the other device to be easily and accurately estimated, even in situations where a general-purpose common positioning system (equivalent to a general-purpose positioning system) such as GPS cannot be used, i.e., outside the general-purpose positioning system area, and without the need to prepare a large amount of external world images within the area to be used as the meeting place in advance. Furthermore, compared to using library images, the present invention uses images from the time of the meeting, which has the advantage of not being affected by changes in the images that may occur between the time the library images were captured and the time of use. Furthermore, in the present embodiment, even if the user moves within the meeting area, past images captured during movement are also referenced, allowing the meeting to be carried out without specifying a specific point within the area.

[0244] (Variation 1) In order to obtain accurate coordinate transformation parameters, the more successfully matched external world feature points are, the better. Therefore, it is desirable that the external world ranges captured by both terminals overlap as much as possible and capture a range in which many distinctive external world points are distributed. For this reason, the terminals determine the quality of the captured range by looking at the distribution of successfully matched external world feature points within the video. When this distribution is in the center of the captured video, it is highly likely that the external world ranges captured by both terminals overlap greatly and capture an external world subject range in which many distinctive external world points that are easy to match are distributed. Therefore, the captured range is updated so that the center of the distribution of matched external world feature points is as close to the center of the captured range as possible. This makes it possible to achieve more accurate position estimation of the other terminal.

[0245] In this embodiment, matching of external feature points is performed between a pair of images taken by the own terminal and the other terminal at a certain location, but the shooting range can be corrected as long as the user does not move too far from the shooting location. This method can also be used in the first embodiment.

[0246] An example of a method for updating the shooting range will be described with reference to Fig. 33. Fig. 33 is a diagram illustrating the processing of Modification 1 of the sixth embodiment.

[0247] In the example of FIG. 33 , the distribution of external world feature points that match the original shooting range with the video captured by the other device is displayed as a superimposed virtual object, but this distribution is biased toward the upper right of the shooting range. In this case, the shooting range is updated toward the upper right and new video is captured so that the center of the external world feature point distribution is closer to the center of the shooting range. Alternatively, to include the target area, shooting may be performed in wide-angle mode, or a camera may be used that is pointed in a direction that includes the target area. This updating of the shooting range may be performed automatically by the device, or a process may be performed in which the user is guided in the shooting direction and prompted to reshoot. Examples of displaying guidance for the shooting direction include the display of the matched external world feature point distribution itself in the original captured video, or the center point of the distribution, superimposed on a virtual object. Note that in the first embodiment, information about the feature point distribution center may be transmitted to the other device, and the other device may be requested to update the shooting range and reshoot.

[0248] Changing the imaging range may change the distribution of matched external feature points. In this case, the new center of the distribution of external feature points may deviate from the center of the imaging range. In this case, the imaging range must be repeatedly updated.

[0249] The termination condition for updating the shooting range may be a certain number of times, with the termination being when the number of matching external feature points no longer increases. After updating the shooting range is completed, a determination process is performed based on the number of successful matches for the video pair with the largest number of matching external feature points.

[0250] (Variation 2) In this embodiment, after the trajectory approaching portion is identified and the coordinate transformation parameters are calculated, the area captured by the other terminal can also be determined. Therefore, when new coordinate transformation parameters are calculated using images from different locations and averaged to improve the accuracy of the coordinate transformation parameters, it is also possible to show the user the location where the other terminal captured images to guide them in taking additional images.

[0251] FIG. 34 is a diagram for explaining the processing of the second modified example of the sixth embodiment, showing an example in which a shooting target is displayed superimposed with a virtual object.

[0252] As shown in the example of FIG. 34, the video captured by the partner terminal at that location may also be displayed.

[0253] (Variation 3) In the above description, the same location is photographed in the trajectory approaching portion, but when photographing the same external area at a distance from the photographing location, the trajectories do not need to be close to each other.

[0254] For example, if the meeting place is a specific floor of a building, before entering the building, video of the building exterior or a familiar location near the building may be captured, and from those videos, video pairs with a certain number of matching external world feature points may be extracted and used to calculate the coordinate transformation parameters. In this case, the external world image captured by the first information terminal is designated as the third external world image, the external world feature points in the third external world image are designated as the third external world feature points, the external world image captured by the second information terminal is designated as the fourth external world image, and the external world feature points in the fourth external world image are designated as the fourth external world feature points. The calculation procedure for the coordinate transformation parameters can be performed in the same manner as above, by replacing the first external world image with the third external world image, the first external world feature points with the third external world feature points, the second external world image with the fourth external world image, and the second external world feature points with the fourth external world feature points. If the location is one where a general-purpose positioning system such as GPS is available, data on the location where the image was captured using the general-purpose positioning system can be sent from the device that first captured the location to the other device, and when the other device captures the image, a guide display of the captured image can be displayed using a virtual object superimposed on it, as shown in Figure 34. In this case, errors in the coordinate transformation parameters may be large due to the distance of the subject, and errors in the autonomous individual positioning using acceleration sensors 115A, 115B and gyro sensors 116A, 116B may increase as the distance traveled from the coordinate transformation parameter determination point increases. However, since the approximate capture position of the other device can be determined when additional images are captured near the meeting place, it is possible to easily capture the same location. Furthermore, even in locations where a general-purpose positioning system is available, it is worthwhile to calculate the coordinate transformation parameters because the effect of errors in the general-purpose positioning system can be reduced.

[0255] (Variation 4) In the above embodiment, the coordinate transformation parameters are determined using a pair of trajectory approaching part images captured at the trajectory approaching part. However, there may be an embodiment in which the first information terminal and the second information terminal each detect the terminal position and capture an external world image, but do not detect the distance to the external world feature point. In this case, index information linking the terminal position at the time of capture with the external world image is also stored in association with the external world image.

[0256] In this modification, a trajectory approaching portion video pair is determined (S01A to S04A and S01B to S04B in FIG. 24, however, distance measurement to external feature points is not performed in S01A and S01B), and index information of frames of the first information terminal included in the trajectory approaching portion video pair is output to the user of the first information terminal, and index information of frames of the second information terminal included in the trajectory approaching portion video pair is output to the user of the second information terminal as meeting support information (S10A, S10B). The frame index information may include, for example, the time the frame was captured. Each terminal can determine the location where the frame was captured from the frame index information.

[0257] (Advantages of this embodiment) According to this embodiment, the user of the first information terminal and the user of the second information terminal can easily find each other by returning to the location where the trajectory approach portion image pair was captured, i.e., the trajectory approach portion. In addition, since coordinate conversion is not performed, the processing load on each information terminal can be reduced.

[0258] Although the embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0259] The programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each process is performed may also be changed.

[0260] Some or all of the functions of the present embodiment described above may be implemented in hardware, for example, by designing them as integrated circuits. Alternatively, they may be implemented in software by a microprocessor unit, CPU, or the like interpreting and executing an operating program that implements each function. Furthermore, the scope of software implementation is not limited, and hardware and software may be used together. Alternatively, some or all of the functions may be implemented in a server. The server may be any type of server, such as a local server, cloud server, edge server, or internet service, as long as it can execute functions in cooperation with other components via communications. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communications network.

[0261] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all of the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

[0262] The above embodiment includes the following aspects.

[0263] (Supplementary Note 1) An information terminal comprising: a processor; a camera; and a communication interface, wherein the camera captures a first external world image within a rendezvous area while the information terminal is moving within the rendezvous area, the processor creates first external world feature point information indicating a relative positional relationship between a first external world feature point in the first external world image and the information terminal based on the first external world image, receives via the communication interface a plurality of second external world images captured by the other information terminal while it is moving within the rendezvous area and a plurality of second external world feature point information indicating a relative positional relationship between second external world feature points in the plurality of second external world images and the other information terminal at the time of capturing the images based on the plurality of second external world images, selects one second external world image from the plurality of second external world images, and compares the first external world image and the first external world feature point information with the selected second external world image and the second external world feature point information corresponding to the selected second external world image to calculate an inter-terminal positional relationship between the information terminal and the other information terminal, The information terminal outputs meeting support information for guiding the user of the information terminal to the terminal position of the other person's information terminal based on the calculated inter-terminal positional relationship.

[0264] (Supplementary Note 2) A meeting support method executed in an information terminal, comprising the steps of: generating first external environment feature point information indicating a relative positional relationship between the information terminal and first external environment feature points in a first external environment image based on a first external environment image within a meeting place area captured by a camera mounted on the information terminal while the information terminal is moving within the meeting place area; receiving from another information terminal communicatively connected to the information terminal a plurality of second external environment images captured within the meeting place area and a plurality of second external environment feature point information indicating a relative positional relationship between the information terminal and second external environment feature points in the plurality of second external environment images based on the plurality of second external environment images; selecting one second external environment image from the plurality of second external environment images; and calculating an inter-terminal positional relationship between the information terminal and the other information terminal by comparing the first external environment image and the first external environment feature point information with the selected second external environment image and the second external environment feature point information corresponding to the selected second external environment image. and outputting rendezvous support information for guiding the user of the information terminal to the location of the other person's information terminal based on the calculated inter-terminal positional relationship.

[0265] (Supplementary Note 3) A meeting support system configured by communicatively connecting an information terminal and another person's information terminal, wherein the information terminal comprises a first camera, a first processor, and a first communication interface, and the other person's information terminal comprises a second camera, a second processor, and a second communication interface, and the second processor of the other person's information terminal creates, based on a plurality of second external world images within a meeting place area captured by the second camera, a plurality of second external world feature point information indicating a relative positional relationship between second external world feature points in the plurality of second external world images at the time of capture and the other person's information terminal, and executes control to transmit the plurality of second external world images and the plurality of second external world feature point information from the second communication interface to the information terminal, and in the information terminal, the first communication interface receives the second external world images and the second external world feature point information, and the first processor a first external world image that indicates a relative positional relationship between the information terminal and a first external world feature point in the first external world image, based on a first external world image within the meeting place area captured by the first camera while the information terminal is approaching the meeting place; a second external world image is selected from the plurality of second external world images; a first external world image is compared with the selected second external world image and the second external world feature point information corresponding to the selected second external world image, the first external world image, and the first external world feature point information to calculate an inter-terminal positional relationship between the information terminal and the other person's information terminal; and a meeting support system that outputs meeting support information for guiding the user of the information terminal to the meeting place, based on the calculated inter-terminal positional relationship.

[0266] 1: Vehicle 1A: In-vehicle terminal 1B: HMD 1C: Smartphone 2: Server 3: Network 4: Access point 100: Meeting support system 111A: Outer camera 111B: Outer camera 111C: Outer camera 112C: Inner camera 113A: Distance measurement sensor 113B: Distance measurement sensor 113C: Distance measurement sensor 115A: Acceleration sensor 115B: Acceleration sensor 115C: Acceleration sensor 116A: Gyro sensor 116B: Gyro sensor 116C: Gyro sensor 117A: Geomagnetic sensor 117B: Geomagnetic sensor 117C: Geomagnetic sensor 118A: Positioning sensor 118B: Positioning sensor 118C: Positioning sensor 119A: Display 119B: Display 119C: Display 120A: Network communication device 120B: Network communication device 120C: Network communication device 121A: Microphone 121B: Microphone 121C: Microphone 122A: Speaker 122B: Speaker 122C: Speaker 123A: Antenna 123B: Antenna 123C: Antenna 124: Touch panel 125A: Processor 125B: Processor 125C: Processor 126A: Program 126B: Program 126C: Program 127A: Data 127B: Data 127C: Data 128A: Memory 128B: Memory 128C: Memory 131C: Telephone network communication device 135A : Position and orientation sensor 135B : Position and orientation sensor 135C : Position and orientation sensor 136A : PP camera 136B : PP camera 136C : PP camera 137A : PP distance measurement sensor 137B : PP distance measurement sensor 137C : PP distance measurement sensor 140A : Bus 140B : Bus 140C : Bus 220 : Network communication device 223 : Antenna 225 : Processor 226 : Program 227 : Data 228 : Memory 240 : Bus1000: First external world image 1001: Second external world image 1010A: First external world feature point 1010B: First external world feature point 1010C: First external world feature point 1011A: Second external world feature point 1011B: Second external world feature point 1011C: Second external world feature point 1200: External world image 1210: Virtual object 1211: Virtual object 1300: External world image 1310: Virtual object 1311: Virtual object

Claims

1. An information terminal comprising: a processor; a camera; and a communication interface, wherein the camera captures a first external world image within a rendezvous area while the information terminal is moving within the rendezvous area, the processor creates first external world feature point information indicating a relative positional relationship between a first external world feature point in the first external world image and the information terminal based on the first external world image, receives via the communication interface a plurality of second external world images captured by the other information terminal while it is moving within the rendezvous area and a plurality of second external world feature point information indicating a relative positional relationship between a second external world feature point in the plurality of second external world images and the other information terminal at the time of capture based on the plurality of second external world images, selects one second external world image from the plurality of second external world images, and compares the first external world image and the first external world feature point information with the selected second external world image and the second external world feature point information corresponding to the selected second external world image to calculate an inter-terminal positional relationship between the information terminal and the other information terminal, The information terminal outputs meeting support information for guiding the user of the information terminal to the terminal position of the other person's information terminal based on the calculated inter-terminal positional relationship.

2. An information terminal according to claim 1, further comprising a distance measurement sensor and a position and orientation sensor, wherein the camera measures an orientation of each of a plurality of first external field feature points included in the first external field image as seen from a terminal position based on a housing of the information terminal, the distance measurement sensor measures a distance from the position of the information terminal to each of a plurality of first external field feature points included in the first external field image, and the position and orientation sensor measures a position and orientation of the information terminal in a first terminal world coordinate system which is a terminal world coordinate system of the information terminal, and the processor calculates coordinate values ​​in a terminal local coordinate system from data on the orientation and distance of each of the plurality of first external field feature points as seen from the information terminal, converts the coordinate values ​​of the plurality of first external field feature points in the terminal local coordinate system into coordinate values ​​in the first terminal world coordinate system from the position and orientation data in the terminal world coordinate system of the information terminal, and creates information indicating a relative positional relationship between the information terminal and each of the plurality of first external field feature points indicated by coordinate values ​​in the first terminal world coordinate system as the first external field feature point information, searching for the second external environment feature points captured in the second external environment images received from the other person's information terminal to obtain a correspondence relationship between the first external environment feature points and the second external environment feature points; selecting a second external environment image in which a number of corresponding second external environment feature points is equal to or greater than a predetermined certain number from among the second external environment images; receiving information indicating a relative positional relationship between the other person's information terminal and each of the second external environment feature points represented by coordinate values ​​in a second terminal world coordinate system that is a terminal world coordinate system of the other person's information terminal as the second external environment feature point information corresponding to the selected second external environment image; comparing the first external environment feature point information with the second external environment feature point information corresponding to the selected second external environment image based on the correspondence relationship between the first external environment feature points and the second external environment feature points; using the conversion information to convert the terminal position of the other party's information terminal expressed in the second terminal world coordinate system into a terminal position expressed in the first terminal world coordinate system, thereby calculating an inter-terminal positional relationship between the information terminal and the other party's information terminal; and generating and outputting the meeting support information based on the calculated inter-terminal positional relationship.Information terminal.

3. An information terminal as described in claim 2, wherein the processor calculates, as the transformation information, a rotation parameter that aligns the orientations of the first terminal world coordinate system and the second terminal world coordinate system from directional vector values ​​in the first terminal world coordinate system and the second terminal world coordinate system of two directional vectors defined in the external world, and calculates a difference between the origins of the first terminal world coordinate system and the second terminal world coordinate system from coordinate values ​​in the first terminal world coordinate system and the second terminal world coordinate system of a position defined in the external world.

4. An information terminal as described in claim 2, wherein the processor generates and outputs the meeting support information based on an average position of estimated positions of multiple other information terminals based on multiple pairs of the first external feature points and the second external feature points that are in the corresponding relationship.

5. An information terminal as described in claim 2, wherein the processor generates and outputs the meeting support information based on transformation information obtained by averaging a plurality of pieces of transformation information calculated from a plurality of sets of the first outside world image at a plurality of locations and a corresponding selected second outside world image.

6. An information terminal as described in claim 2, further comprising a display, wherein the processor, after determining the conversion information, displays on the display a virtual object indicating the subject position of the second outside world image captured by the other person's information terminal, superimposed on the outside world image within the meeting place area.

7. An information terminal as described in claim 1, further comprising a display, and a virtual object indicating a distribution of the first external world feature points and the second external world feature points that have a corresponding relationship between the first external world image and the second external world image is displayed on the display superimposed on the external world image within the meeting place area.

8. An information terminal as described in claim 1, further comprising a display, wherein the processor displays, as the meeting support information, a virtual object indicating a route along a movement route of the information terminal or the other person's information terminal to an estimated terminal position of the other person's information terminal, superimposed on an external world image within the meeting place area on the display.

9. An information terminal as described in claim 1, in a general-purpose positioning system area having a general-purpose positioning system that can be commonly used by the information terminal and the other information terminal, wherein the camera captures a third external world image in the general-purpose positioning system area, the processor creates third external world feature point information indicating a relative positional relationship between a third external world feature point in the third external world image and the information terminal based on the third external world image, receives via the communication interface a fourth external world image captured by the other information terminal in the general-purpose positioning system area and a plurality of fourth external world feature point information indicating a relative positional relationship between a fourth external world feature point in the fourth external world image and the positional relationship of the other information terminal at the time of capture based on the fourth external world image, and compares the third external world image and the third external world feature point information with the fourth external world image and the fourth external world feature point information to calculate the inter-terminal positional relationship between the information terminal and the other information terminal.

10. A meeting support method executed in an information terminal, comprising the steps of: generating first external environment feature point information indicating a relative positional relationship between the information terminal and a first external environment feature point in a first external environment image based on a first external environment image in a meeting place area captured by a camera mounted on the information terminal while moving within the meeting place area; receiving from a third party information terminal communicatively connected to the information terminal a plurality of second external environment images captured within the meeting place area and a plurality of second external environment feature point information indicating a relative positional relationship between the information terminal and a second external environment feature point in the plurality of second external environment images based on the plurality of second external environment images; selecting one second external environment image from the plurality of second external environment images; and calculating an inter-terminal positional relationship between the information terminal and the third party information terminal by comparing the first external environment image and the first external environment feature point information with the selected second external environment image and the second external environment feature point information corresponding to the selected second external environment image. outputting rendezvous support information for guiding the user of the information terminal to the position of the other person's information terminal based on the calculated inter-terminal positional relationship.

11. A meeting support system configured by communicatively connecting an information terminal and an other person's information terminal, wherein the information terminal comprises a first camera, a first processor, and a first communication interface, and the other person's information terminal comprises a second camera, a second processor, and a second communication interface, and the second processor of the other person's information terminal creates a plurality of second external environment feature point information indicating a relative positional relationship between second external environment feature points in the plurality of second external environment images at the time of capture and the other person's information terminal based on a plurality of second external environment images within a meeting place area captured by the second camera, and executes control to transmit the plurality of second external environment images and the plurality of second external environment feature point information from the second communication interface to the information terminal, and in the information terminal, the first communication interface receives the second external environment images and the second external environment feature point information, and the first processor a first external world image that is captured by the first camera while the information terminal is approaching the meeting place, the first external world image being used to generate first external world feature point information indicating a relative positional relationship between the information terminal and a first external world feature point in the first external world image, the first external world image being used to compare the selected second external world image and the second external world feature point information corresponding to the selected second external world image with the first external world image and the first external world feature point information, the second external world image being used to calculate an inter-terminal positional relationship between the information terminal and the other person's information terminal, and the third person's information terminal being used to output meeting support information that guides the user of the information terminal to the meeting place, based on the calculated inter-terminal positional relationship.

Images