Information terminal, meetup support method, and meetup support system

Abstract

This information terminal comprises a processor, a camera, and a communication interface. The processor: creates, on the basis of a first external video captured in the vicinity of a meeting place by the camera while approaching the vicinity of the meeting place, first external feature point information indicating a relative positional relationship between a first external feature point in the first external video and the information terminal; receives, via the communication interface, a second external video captured in the vicinity of the meeting place by an another person's information terminal, and second external feature point information indicating a relative positional relationship between a second external feature point in the second external video and said another person's information terminal; compares the first external video and the second external video, and the first external feature point information and the second external feature point information to calculate an inter-terminal positional relationship between the information terminal and said another person's information terminal; and outputs, on the basis of the calculated inter-terminal positional relationship, meetup support information for guiding the user of the information terminal to the meeting place.

Description

Information terminal, meeting support method, and meeting support system

[0001] The present invention relates to an information terminal, a meeting support method, and a meeting support system.

[0002] As a technology for supporting meetings, the information transmission / reception device in Patent Document 1 is described as comprising "a current location acquisition means provided on the side of a first user who is meeting a second user at a predetermined meeting place, and which includes: a current location acquisition means for acquiring the current location of the first user; a meeting place acquisition means for acquiring the meeting place for the first user and the second user; a traveling direction prediction means for predicting the traveling direction of the first user when arriving at the meeting place based on the positional relationship between the current location of the first user and the meeting place; an imaging command generation means for generating an imaging command for capturing an image of the meeting place in a predetermined imaging direction based on the traveling direction predicted by the traveling direction prediction means; and a transmission / reception means for sending the generated imaging command to the second user (abstract excerpt)."

[0003] Japanese Patent Application Laid-Open No. 2005-17074

[0004] In the information transmission and reception device of Patent Document 1, a user heading to a meeting place by car can tell that they have arrived near the meeting place by comparing the image taken by the other user at the meeting place with the scenery in front of the car, but there is a problem in that they cannot know the exact location of the other user.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an information terminal, a meeting support method, and a meeting support system that accurately notify a user of the location of a target location such as a meeting place.

[0006] In order to solve the above-mentioned problems, the present invention has the configuration described in the claims. For example, an information terminal includes a processor, a camera, and a communication interface, wherein the camera captures a first external world image of a vicinity of a meeting place while the information terminal is approaching the vicinity of the meeting place, the processor generates first external world feature point information indicating a relative positional relationship between a first external world feature point in the first external world image and the information terminal based on the first external world image, receives via the communication interface a second external world image captured by a third party's information terminal of the vicinity of the meeting place and second external world feature point information indicating a relative positional relationship between a second external world feature point in the second external world image and the third party's information terminal, compares the first external world image with the second external world image and the first external world feature point information with the second feature point information to calculate an inter-terminal positional relationship between the information terminal and the third party's information terminal, and outputs meeting support information for guiding the user of the information terminal to the meeting place based on the calculated inter-terminal positional relationship.

[0007] According to the present invention, it is possible to provide an information terminal, a meeting support method, and a meeting support system that accurately notify a user of the location of a target location such as a meeting place. Note that problems, configurations, and effects other than those described above will become clear from the description of the following embodiments.

[0008] 1 is a configuration diagram of a meeting support system according to an embodiment of the present invention. FIG. 2 is a hardware configuration diagram of an information terminal. FIG. 3 is a hardware configuration diagram of a server. FIG. 4 is a diagram illustrating an example of usage of the meeting support system according to an embodiment of the present invention. FIG. 5 is a diagram illustrating an image captured from a user information terminal. FIG. 6 is a diagram illustrating an image captured from an other person's information terminal. FIG. 7 is a diagram illustrating an example of external world feature point information. FIG. 8 is a diagram illustrating an example of a correspondence relationship between external world feature points. FIG. 9 is a diagram illustrating an example of measuring coordinate values ​​of external world feature points in a terminal world coordinate system. FIG. 10 is a diagram illustrating an example of evaluating feature quantities of external world feature points. FIG. 11 is a diagram illustrating the contents of a method (1) for acquiring an inter-terminal positional relationship between an other person's information terminal and a user information terminal, and is an explanatory diagram of two directional vectors defined in real space. FIG. 12 is a diagram illustrating the contents of a method (1) for acquiring an inter-terminal positional relationship between an other person's information terminal and a user information terminal, and is a diagram illustrating the relationship between the numerical expressions of two directional vectors in the terminal world coordinate systems of the user information terminal and the other person's information terminal. FIG. 13 is a diagram illustrating a concept of separating the positional relationship between an external world feature point and an HMD 1B, which is the other person's information terminal, into a horizontal projection plane and a vertical projection plane. 13A and 13B are diagrams showing the contents of a method (2) for acquiring the inter-terminal positional relationship of an other information terminal with respect to a user information terminal, and showing a method for calculating coordinate values ​​of the other information terminal in the coordinate system of the user information terminal in a projection plane. FIG. 13B are diagrams showing the contents of a method (2) for acquiring the inter-terminal positional relationship of an other information terminal with respect to a user information terminal, and showing a method for calculating coordinate values ​​of the other information terminal in the coordinate system of the user information terminal in three-dimensional space. FIG. 13C are diagrams showing examples of processing for special situations in the processing of FIG. 13A. FIG. 13C are diagrams showing the contents of a method (2) for acquiring the inter-terminal positional relationship of an other information terminal with respect to a user information terminal, and showing an explanatory diagram of two directional vectors defined in real space. FIG. 13D are diagrams showing an example of changing to a low-distortion shooting mode when shooting an outside world image. FIG. 13D are diagrams showing an example of changing to a wide-angle shooting mode when shooting an outside world image. FIG. 13E are diagrams showing an example of displaying an image shot by a partner terminal when shooting an outside world image. FIG. 13F are diagrams showing an example of displaying a virtual object at a shooting target point when shooting an outside world image. FIG. 13F are diagrams showing an example of displaying a prompt to take additional photos when shooting an outside world image. 10A and 10B are diagrams illustrating an example in which, when capturing an external image, the partner terminal displays a message indicating that additional capture is not possible.FIG. 10 is a diagram showing an example of information display on a user information terminal (vehicle-mounted terminal 1A) based on terminal position estimation for meeting support. FIG. 11 is a diagram showing an example of information display on an other person's information terminal (HMD 1B) based on terminal position estimation for meeting support. FIG. 12 is a flowchart showing the processing flow of the meeting support system according to the present embodiment. FIG. 13 is a diagram showing an example of setting external feature points in the second embodiment. FIG. 14 is a diagram showing an example of the relationship between a shooting point and a meeting place in the fourth embodiment.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same components are designated by the same reference numerals throughout the drawings, and redundant explanations will be omitted.

[0010] In this embodiment, the in-vehicle terminal mounted on a vehicle and the portable information terminal carried by a user are collectively referred to as the information terminal. When a user uses a portable information terminal while on a vehicle, it corresponds to the in-vehicle terminal.

[0011] The information terminal according to this embodiment is characterized in that a processor installed in a terminal (hereinafter referred to as an "information terminal") used by a user heading to a meeting place calculates a terminal-to-terminal positional relationship between the information terminal and the other person's information terminal present at a destination point, using information relating to an external world image and external world feature points in the external world image acquired from an information terminal operated by another person (hereinafter referred to as an "other person's information terminal"), and information relating to an external world image captured by a camera installed in the mobile information terminal and external world feature points in the external world image, and outputs guidance information indicating the destination point to the mobile information terminal based on the terminal-to-terminal positional relationship. Therefore, the present invention can increase the commercial value of information processing devices to which the present invention is applied, and is therefore expected to contribute to Goal 8.2 of the Sustainable Development Goals (SDGs) proposed by the United Nations ("SDGs") ("Increase economic productivity through diversification, technological improvement, and innovation, particularly in industries that increase the value of goods and services and in labor-intensive industries").

[0012] The following describes an embodiment of the present invention, exemplifying an embodiment in which the present invention is applied to a meeting support system between a user of an information terminal and another person with whom the user has a meeting. Note that the information terminal may be simply referred to as a terminal. While approaching the meeting place, the information terminal user uses the information terminal to capture an external image of the area near the meeting place from a short distance away. The information terminal then generates external world feature point information indicating the relative positional relationship between the external world feature points in the external world image and the information terminal. Meanwhile, the other person uses the other person's information terminal, generally at the meeting place, to capture an external world image of the area near the meeting place. The other person's information terminal then generates external world feature point information indicating the relative positional relationship between the external world feature points in the external world image and the other person's information terminal. The external world images and external world feature point information of the two terminals are compared to determine the inter-terminal positional relationship, which is the relative positional relationship between the two terminals. The information terminal then guides the user to the meeting place based on this inter-terminal positional relationship information. In this case, the meeting place corresponds to the destination point to which the user of the information terminal is heading. The destination point is stationary in the world coordinate system. The user moves toward the destination point together with the information terminal.

[0013] First Embodiment FIG. 1 is a configuration diagram of a meeting support system 100 according to this embodiment.

[0014] An example of an information terminal is an in-vehicle terminal 1A. While approaching the meeting place, the in-vehicle terminal 1A captures an external image of the vicinity of the meeting place and generates external feature point information indicating the relative positional relationship between the external feature points in the external image and the information terminal. The other person waiting at the meeting place carries an HMD 1B or a smartphone 1C, which is an information terminal of the other person.

[0015] The in-vehicle terminal 1A, the HMD 1B, and the smartphone 1C are connected to one another via a network 3. Furthermore, the in-vehicle terminal 1A, the HMD 1B, and the smartphone 1C are connected to a server 2 via the network 3. The server 2 is a server that provides a meeting support service, and may be a cloud server or an edge server.

[0016] The other person uses the other person's information terminal to capture an external world image near the meeting place, creates external world feature point information indicating the relative positional relationship between the external world feature points in the external world image and the other person's information terminal, and transmits this information to the vehicle-mounted terminal 1A. The vehicle-mounted terminal 1A determines the other person's position at the meeting place using the received external world image and external world feature point information, the external world image captured by the vehicle-mounted terminal 1A, and the created external world feature point information.

[0017] The in-vehicle terminal 1A includes a processor connected to in-vehicle equipment such as an in-vehicle camera, a car navigation system, and a GPS mounted on the vehicle 1. For example, the in-vehicle terminal 1A may be configured by connecting a smartphone operated by a passenger in the vehicle 1, such as the driver, to in-vehicle equipment such as an in-vehicle camera using Bluetooth (registered trademark). The in-vehicle terminal 1A may also be configured by connecting the in-vehicle equipment and the processor via a vehicle network (CAN). Furthermore, the in-vehicle terminal 1A may be configured by a standalone mobile information terminal such as a smartphone operated by a passenger in the vehicle 1.

[0018] The information terminal may be an HMD 1B or a smartphone 1C. That is, the information terminal refers to a terminal carried by a user who is moving toward a target location, which is a meeting place, and the type of device used may be the in-vehicle terminal 1A, the HMD 1B, or the smartphone 1C.

[0019] Similarly, the other person's information terminal means a terminal for specifying a target point, and is not limited to the HMD 1B or the smartphone 1C, but may also be the in-vehicle terminal 1A.

[0020] FIG. 2 is a diagram showing the hardware configuration of the information terminal.

[0021] 2 shows the components necessary to realize the functions of the information terminal according to this embodiment and the components that are often typically installed. Furthermore, since the configurations of the information terminals used in this embodiment are similar to each other, they are described together. The configurations of the in-vehicle terminal 1A, HMD 1B, and smartphone 1C are distinguished by reference numerals A, B, and C, respectively. The in-vehicle terminal 1A, HMD 1B, and smartphone 1C may be collectively referred to as the terminal.

[0022] The in-vehicle terminal 1A, the HMD 1B, and the smartphone 1C include outer cameras 111A, 111B, and 111C, distance measurement sensors 113A, 113B, and 113C, position and orientation sensors 135A, 135B, and 135C, geomagnetic sensors 117A, 117B, and 117C, positioning sensors 118A, 118B, and 118C, and a real-time clock (RTC) The components include clocks 114A, 114B, and 114C, displays 119A, 119B, and 119C, microphones 121A, 121B, and 121C, speakers 122A, 122B, and 122C, processors 125A, 125B, and 125C, memories 128A, 128B, and 128C, and network communicators 120A, 120B, and 120C, which are connected to one another via buses 140A, 140B, and 140C that connect the components together.

[0023] Specific examples of the distance measurement sensors 113A, 113B, and 113C include a TOF (Time of Flight) sensor and a LiDAR (Laser Imaging Detection and Ranging) sensor.

[0024] The network communicators 120A, 120B, 120C are connected to antennas 123A, 123B, 123C that transmit and receive network communication signals.

[0025] The displays 119A, 119B, and 119C and the speakers 122A, 122B, and 122C correspond to output devices, and the microphones 121A, 121B, and 121C correspond to input devices.

[0026] The position and orientation sensors 135A, 135B, and 135C are used to measure the position and orientation of the terminal. Here, orientation refers to the orientation of the terminal in the real space in which the terminal exists. Specific configuration examples of the position and orientation sensors 135A, 135B, and 135C may include position posture cameras 136A, 136B, and 136C (hereinafter abbreviated as PP cameras), position posture distance measurement sensors 137A, 137B, and 137C (hereinafter abbreviated as PP distance measurement sensors), acceleration sensors 115A, 115B, and 115C, and gyro sensors 116A, 116B, and 116C.

[0027] The PP cameras 136A, 136B, and 136C may be the same as the outer cameras 111A, 111B, and 111C.

[0028] Furthermore, the PP distance measuring sensors 137A, 137B, and 137C may be the same as the distance measuring sensors 113A, 113B, and 113C.

[0029] Feature points that exist in the external world (hereinafter referred to as "external world feature points") are used to measure the position and orientation using the PP cameras 136A, 136B, and 136C and the PP ranging sensors 137A, 137B, and 137C. A plurality of external world feature points are used, and for example, external world feature points inside a room or external world feature points on a building outdoors may be used. As a preparation, the coordinate values ​​of the external world feature points are determined from the direction and distance measurements of the external world feature points. If the coordinate values ​​of the external world feature points in the external world coordinate system are known, this preparation can be omitted.

[0030] Then, by measuring changes in the direction and distance of the external feature points as seen from the terminal, changes in the position and orientation of the terminal from the relative positional relationship with the initial external feature points are measured. When measuring distance using a stereo camera method, only PP cameras 136A, 136B, and 136C may be used.

[0031] Note that while grasping the vertical direction is important in practical applications, if the coordinate values ​​of the external feature points in the external coordinate system (world coordinate system) and the vertical direction in the external coordinate system are known, the relative positional relationship between the terminal housing attitude and the vertical direction can be grasped by measuring the external feature points using PP cameras 136A, 136B, and 136C and PP ranging sensors 137A, 137B, and 137C. That is, for example, by measuring the coordinate values ​​of three external feature points in the terminal local coordinate system, the relationship between the numerical representations of two directional vectors between the external feature points in the external coordinate system and the terminal local coordinate system can be determined, and a rotation transformation that aligns the orientation of the terminal local coordinate system with the external coordinate system can be obtained. This determines the relationship between the terminal housing and the vertical direction.

[0032] Measurement of position and attitude using acceleration sensors 115A, 115B, and 115C and gyro sensors 116A, 116B, and 116C is performed by integrating acceleration and angular acceleration. This allows for determination of displacement from a reference position and reference attitude. Furthermore, measurement of the gravitational acceleration vector by acceleration sensors 115A, 115B, and 115C also allows for determination of the terminal's housing attitude and its relative position in the vertical direction.

[0033] The position and attitude of the terminal may be measured by referring to both the measurements using the PP cameras 136A, 136B, 136C and the PP ranging sensors 137A, 137B, 137C and the measurements using the acceleration sensors 115A, 115B, 115C and the gyro sensors 116A, 116B, 116C, or by using measurements from other types of sensors.

[0034] Positioning sensors 118A, 118B, and 118C are used to measure the indoor and outdoor positions of terminals. Specific examples of indoor positioning sensors include those that use beacon signals and location indicator marks. Outdoor positioning sensors include, for example, GPS (Global Positioning System) receivers that use signals from satellites.

[0035] The network communicators 120A, 120B, and 120C are communication interfaces that perform wireless communication between at least the terminal and the access point 4 via short-range wireless communication, wireless LAN, or base station communication, and include communication processing circuits corresponding to various predetermined communication interfaces and are connected to the antennas 123A, 123B, and 123C. The network communicators 120A, 120B, and 120C transmit and receive biometric information, video data, control signals, and the like. Note that short-range wireless communication is performed using a wireless LAN such as Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or Wi-Fi (registered trademark). Furthermore, for base station communication, long-distance wireless communication such as LTE (Long Term Evolution, registered trademark), LTE-Advanced, Mobile WiMAX (Worldwide Interoperability for Microwave Access, registered trademark), or WiMAX2 may be used.

[0036] Displays 119A, 119B, and 119C are installed in a manner that allows the terminal users to view images. Processors 125A, 125B, and 125C control the operation of the terminals.

[0037] The processors 125A, 125B, and 125C are configured by, for example, a CPU.

[0038] The memories 128A, 128B, and 128C are configured by flash memory and nonvolatile memory. The memories 128A, 128B, and 128C also store programs 126A, 126B, and 126C such as an OS (Operating System) and operation control applications, and data 127A, 127B, and 127C used by the processors 125A, 125B, and 125C.

[0039] The processors 125A, 125B, 125C load the programs 126A, 126B, 126C into the memories 128A, 128B, 128C and execute them, and read out the data 127A, 127B, 127C as needed and use it in the execution processing of the programs 126A, 126B, 126C.

[0040] The smartphone 1C further includes, as components, an inner camera 112C, a touch panel 124 as an input interface, and a telephone network communication device 131C.

[0041] FIG. 3 is a diagram showing the hardware configuration of the server.

[0042] In the server 2, a processor 225, a memory 228, and a network communicator 220 are connected to one another via a bus 240 that connects the various components. The network communicator 220 is connected to an antenna 223 that transmits and receives network communication signals.

[0043] The processor 225 controls the operation of the server and is configured by, for example, a CPU.

[0044] The memory 228 is configured by a flash memory and a nonvolatile memory, and also stores programs 226 such as an OS (Operating System) and an operation control application, and data 227 used by the processor 225.

[0045] The processor 225 loads the program 226 into the memory 228 and executes it, and reads out the data 227 as necessary to use it in the execution process of the program 226 .

[0046] [Processing Flow] FIG. 4 is a diagram showing an example of the usage status of the meeting support system 100 according to this embodiment.

[0047] Assume that a passenger and a taxi are meeting at the location where the passenger is standing in the figure. The passenger corresponds to a third party who specifies the destination point, which is the meeting place. The taxi corresponds to vehicle 1 in Figure 1, and the taxi driver corresponds to a user traveling toward the destination point, which is the meeting place.

[0048] The taxi is equipped with an in-vehicle terminal 1A, and the passenger carries an HMD 1B. The in-vehicle terminal 1A corresponds to the user information terminal. The HMD 1B corresponds to the other person's information terminal. The following process is basically the same even if the other person's information terminal is a smartphone 1C. To know the relative positional relationship between the terminals, a video of the vicinity of the meeting place is taken. The video may be a video or a still image. The relative positional relationship between the terminals may be referred to as the inter-terminal positional relationship.

[0049] Fig. 5 is a diagram showing an image captured from a user information terminal, and Fig. 6 is a diagram showing an image captured from another person's information terminal. The first external world image 1000 shown in Fig. 5 includes first external world feature points 1010A, 1010B, and 1010C. Similarly, the second external world image 1001 shown in Fig. 6 includes second external world feature points 1011A, 1011B, and 1011C. Because it is necessary for both terminals to capture roughly the same location, approximate data on the subject position, shooting location, and shooting direction may be communicated from one terminal to the other terminal or bidirectionally using positioning data from a general-purpose positioning system such as GPS that can determine the approximate position.

[0050] (Method for acquiring the positional relationship between terminals (1)) First, we will explain the terms related to coordinate systems. First, a "world coordinate system" is a coordinate system that is stationary relative to the external world, and a "local coordinate system" is a coordinate system that is not stationary relative to the external world but is fixed to some kind of object. Then, a "terminal world coordinate system" is a coordinate system that is fixed relative to the external world and is used by the terminal under consideration, and a "terminal local coordinate system" is a coordinate system that is fixed to the housing of the terminal under consideration.

[0051] Each terminal requires some kind of "world coordinate system" to describe its own position and orientation relative to the external world as its control data. The "world coordinate system" used by each terminal is called the "terminal world coordinate system." Generally, the "terminal world coordinate system" differs from terminal to terminal. The core of the present invention is a technology described in "Methods (1) and (2) for Acquiring Positional Relationship Between Terminals," which compares external world images captured by two terminals and determines coordinate transformation parameters between the two terminal world coordinate systems. These coordinate transformation parameters determine the position coordinates of the other terminal within the terminal world coordinate system, or vice versa. The difference between acquisition methods (1) and (2) is the accompanying information of the external world image. In using external world images, the present invention falls under the category of VPS (Visual Positioning System) technology. However, the present invention differs from conventional VPS technology in that the amount of image information required to meet the objectives of each embodiment is small.

[0052] The "terminal local coordinate system" is used to describe the positions of external feature points and the direction as seen from the terminal, based on the terminal's position and orientation.

[0053] In the explanation, we sometimes say "numerical expression in a coordinate system." We also sometimes shorten it to "expression in a coordinate system," but in this case "expression" means "numerical expression." Specifically, when we say "numerical expression" of a point or position in the outside world, we are referring to the coordinate value of the point or position in a certain coordinate system. If we express it in terms of components for each coordinate axis in a three-dimensional coordinate system, it is as follows: (P Ｘ , P Ｙ , P Ｚ ) is data such as this. Direction, or the "numerical expression" of a directional vector, which is the same thing, is a numerical representation of a directional vector in a certain coordinate system. If expressed in components for each coordinate axis in a three-dimensional coordinate system, it is expressed as (V Ｘ , V Ｙ , V Ｚ) The specific values ​​of these "numerical representations" change when the coordinate system changes, even if they represent the same point or the same directional vector. In the process described below, the specific values ​​of the "numerical representation" of the same point or the same directional vector change depending on the respective "terminal world coordinate systems," so conversion parameters are found to align the "numerical representation" between different "terminal world coordinate systems" based on these differences. Knowing the position of the other terminal means knowing the "numerical representation" of the other terminal's position in the "terminal world coordinate system" of the own terminal.

[0054] Next, we will explain the symbols. In the following explanation, unless otherwise specified, when rotation, direction vectors, and vectors (including point position indications) are represented by a single symbol, they will be represented as quaternions. In addition, the norm of both rotation and direction vectors is normalized to 1. Furthermore, when a direction vector or vector is represented by a quaternion, the real part of the quaternion is set to 0. We will first organize the symbols used in the explanation.

[0055] First, the basis elements of a quaternion other than 1 are represented as i, j, and k, and unless otherwise specified, when representing a direction vector or vector, the relationship to the axis directions of the coordinate system under consideration is X axis: i, Y axis: j, and Z axis: k. For example, when the basis element i in quaternion notation represents a vector, this vector means (1, 0, 0) in the component notation for each of the coordinate axes X, Y, and Z. In each three-dimensional coordinate system, the coordinate axes are represented by a combination of the letters X, Y, and Z and their subscripts.

[0056] A quaternion Q that rotates a direction vector n by an angle λ in the direction of a right-handed screw. Ｒ (n, λ) is expressed by the following formula: Q Ｒ (n, λ) = cos(λ / 2) + sin(λ / 2)n (1) Note that rotating in the direction in which a right-handed screw advances means rotating in the direction in which the screw rotates when a right-handed screw advances (clockwise when looking in the direction in which the screw advances).

[0057] Here, Q ＊ is the conjugate quaternion of Q, the order of multiplication when vector a is rotated to vector b by rotation Q is as follows: b = QaQ ＊ ... (2)

[0058] Specifically, the quaternion that rotates 90 degrees around the direction vector n in the direction of n, in the direction of a right-handed screw, is expressed as H(n). H(n) is expressed as follows: H(n) = 2 －１／２ +2 －１／２ n... (3)

[0059] [·] is an operator that normalizes a non-zero quaternion to norm 1. If q is any non-zero quaternion, [·] is defined as follows: [q] = q(qq ＊ ) －１／２ ...(4)

[0060] P Ｔ (n) is an operator that projects a vector onto a plane perpendicular to the direction vector n. n is assumed to be normalized. The projection of vector p is expressed by the following equation: P Ｔ (n) p = p + nS Ｃ (np) (5) Here, Sc(q) denotes the real part of the quaternion q.

[0061] Together with the above symbols, the direction vector n １ and direction vector n ２ Rotate around an axis perpendicular to the plane containing n １ to n ２ Rotation Q to be superimposed １２ Use equation (6) to find Q １２ = [1-n ２ n １ ]・・・(6)

[0062] It can be easily confirmed that the formula (6) holds true by directly calculating the right-hand side, taking care to set the real part of the direction vector to 0.

[0063] Now, in order to match the external feature points in the external image of the area around the meeting point taken by each device, external feature point information for each image is created. The image can be a still image or a video. In the case of a video, at least one frame of the image is extracted and used.

[0064] The external world feature point information is list information that includes the positions of the external world feature points in the video and coordinate value data in the terminal's world coordinate system, corresponding to the captured video. An example of the external world feature point information is shown in FIG. 7A. The coordinate values ​​in FIG. 7A are written in normal component notation. Such external world feature point information is created for each of the videos from both terminals. The coordinate values ​​of the matched external world feature points may be obtained later using the feature point information or external world feature point matching, which will be described later.

[0065] When using the method (2) for acquiring the positional relationship between terminals described later, one of the terminals uses the value of a direction vector in the terminal world coordinate system pointing from the terminal to the external world feature point in place of the coordinate value of the external world feature point in the external world feature point information.

[0066] Here, the position on the video refers to coordinates on the video, for example, coordinates with the upper left corner of the video as the origin, the horizontal direction with the right direction as the positive direction as the U coordinate, and the vertical direction with the downward direction as the positive direction as the V coordinate ( FIG. 8 ). In the following, the coordinate system refers to a world coordinate system fixed to the external world unless otherwise specified. The external world feature information also includes the feature amounts of the external world feature points, which will be described later. Using these feature amounts as clues, external world feature point matching, which will be described later, is performed to determine the correspondence between the external world feature points in the video of both devices. An example of an external world feature point matching table that summarizes this correspondence is shown in FIG. 7B .

[0067] First, as a method (1) for acquiring the positional relationship between terminals, a case will be described in which the external world feature point information of both terminals contains coordinate value data of the external world feature point. To obtain the coordinate value of the external world feature point in the terminal world coordinate system, first, the outer cameras 111A and 111B measure the direction vector of the external world feature point as seen from the terminal position relative to the terminal housing, and the ranging sensors 113A and 113B measure the distance from the terminal position to the external world feature point to obtain the coordinate value in the terminal local coordinate system. Next, the position and orientation sensors 135A and 135B measure the position and orientation of the terminal in the world coordinate system, and the coordinate value in the terminal local coordinate system is converted to a coordinate value in the terminal world coordinate system, thereby obtaining the distance. Distance measurement may also be performed using a moving stereo method using the outer cameras 111A and 111B.

[0068] Here, the method for calculating the coordinate values ​​of the external feature points will be described in detail. First, the terminal local coordinate system {X Ｌ Axis, Y Ｌ axis, Z Ｌ Axis, origin: O Ｌ} is defined (Fig. 8). Then, the unit vectors in the directions of the coordinate axes are defined as N ＸＬ , N ＹＬ , N ＺＬ The specific notation in quaternions is N ＸＬ = i, N ＹＬ = j, N ＺＬ = k, where i, j, and k are the basis elements of the quaternion. The origin of the terminal local coordinate system is the point where the direction of the incident light from the outside world converges. The optical axis direction is X Ｌ axis, and the horizontal direction of the image is perpendicular to the optical axis and points leftward in the Y direction. Ｌ Axis, vertical direction of the image is Z Ｌ The axis is X. Ｌ The axis passes through the center of the image. Ｗ Axis, Y Ｗ axis, Z Ｗ Axis, origin: O Ｗ} is defined to recognize the position and orientation of the terminal. As an example of the definition of the terminal world coordinate system, there is a method of fixing the terminal local coordinate system at the time of terminal initialization to the external world as it is and using it as the terminal world coordinate system. The maximum value of the video coordinate value is (U Ｍ , V Ｍ ) and the horizontal angle of view is μ Ｕ , vertical angle of view is μ Ｖ , the image plane is perpendicular to the optical axis, and distortion caused by the optical system is also corrected.

[0069] Now, the position of the external feature point on the image (U Ｐ , V Ｐ ) The origin of the terminal local coordinate system O Ｌ The distance from the center of the image to the center of the image is f, and the coordinate value on the image (U Ｐ , V Ｐ ) is the position of the point on the image represented by P Ｎ Then, P Ｎ The three-dimensional expression of P in the terminal local coordinate system is expressed by the following equation: Ｎ = fN ＸＬ +fS Ｎ ... (7)

[0070] Here, S Ｎ is expressed as follows in three-dimensional terms in the terminal local coordinate system: Ｎ = ((U Ｍ -2U Ｐ ) / 2U Ｍ ) tan(μ Ｕ / 2) N ＹＬ +((V Ｍ -2V Ｐ ) / 2V Ｍ ) tan(μ Ｖ / 2) N ＺＬ ...(8)

[0071] From this, the origin of the terminal local coordinate system O Ｌ The external feature point direction vector N Ｒ is given by the following equation in numerical expression in the terminal local coordinate system: Ｒ = [N ＸＬ +S Ｎ ]・・・(9)

[0072] The external feature point direction vector N Ｒ and the distance d from the terminal position to the external feature point, which is calculated separately. Ｒ multiplied by , the numerical representation R of the coordinates of the external feature points in the terminal local coordinate system is obtained. Ｌ is obtained. R Ｌ = d Ｒ N Ｒ ...(10)

[0073] Representation of external feature point coordinate values ​​in the device local coordinate system R Ｌ From the expression R in the terminal world coordinate system Ｗ To convert to this, information on the orientation and origin position of the terminal local coordinate system in the terminal world coordinate system is used. The orientation and origin position of the terminal local coordinate system are known because the changes from the initialized state are measured by the position and orientation sensors 135A and 135B. The rotation of the terminal local coordinate system in the terminal world coordinate system from the state where the directions of the X axes, Y axes, and Z axes are aligned to the current point in time is expressed as Q. ＬＷ The coordinate value of the origin of the terminal local coordinates in the terminal world coordinate system is O ＬＷ Then, R Ｗ is calculated as follows: R Ｗ = QＬＷ R Ｌ Q ＬＷ ＊ +O ＬＷ ...(11)

[0074] In the method (2) for acquiring the positional relationship between terminals, which will be described later, the external feature point information of one terminal uses a direction vector from the terminal position to the external feature point instead of the coordinate value of the external feature point. The numerical expression in the terminal world coordinate system of the direction vector from the terminal position to the external feature point is Ｗ Then, N Ｗ is calculated as follows: N Ｗ = Q ＬＷ N Ｌ Q ＬＷ ＊ ...(11')

[0075] Examples of external field feature points are shown in Figures 5 and 6. Here, the corners of window frames of three buildings are extracted as external field feature points as examples of stationary objects. To determine the positional relationship between terminals, three external field feature points that are not on the same line are required. Since there is no guarantee that individual external field feature points can be matched between external field images captured by both terminals with 100% probability, it is desirable to list three or more external field feature points in the external field feature point information. In Figures 5 and 6, the matched external field feature points are shown as first external field feature points 1010A, 1010B, and 1010C and second external field feature points 1011A, 1011B, and 1011C.

[0076] 5 is an example of an external world image captured by a user information terminal, which is designated as a first external world image 1000, and FIG. 6 is an example of an external world image captured by another person's information terminal, which is distinguished from a second external world image 1001. First external world feature points 1010A, 1010B, and 1010C are captured in the first external world image 1000, and second external world feature points 1011A, 1011B, and 1011C are captured in the second external world image 1001. Here, for example, an external world feature point captured in the first external world image is referred to as a first external world feature point, but if there is no need to specify in which image the referenced external world feature point was captured, the expressions specifying the image, such as first or second, may be omitted.

[0077] The distinction between first and second coordinate systems for the external world image may also be used for the terminal coordinate system. That is, the world coordinate system used by the user information terminal may be referred to as the first terminal world coordinate system, and the world coordinate system used by the other person's information terminal may be referred to as the second terminal world coordinate system.

[0078] Here, a method for matching external feature points in the external image of both terminals will be described. First, external feature points, mainly corners such as window frames of buildings, are extracted from the external image. Next, feature quantities of the extracted external feature points are calculated from the external image. There are various methods for calculating feature quantities. For example, SIFT (Scale-Invariant Feature Transform), SURF (Speed-Up Robust Features), BRIEF (Binary Robust Independent Elementary Features), etc., and any feature quantity may be used for matching.

[0079] Here, an overview of SIFT features is explained (Fig. 9). First, the maximum gradient direction of the brightness gradient near the external feature point is found, and the external image area is divided into 16 4x4 blocks near the feature point according to this direction. Then, the brightness gradient in each block is represented as a histogram in eight directions. The data of this histogram is collected and represented as a 128-dimensional vector value, which is the SIFT feature.

[0080] Next, an example of a procedure for matching feature points based on feature amounts will be described. While Figs. 5 and 6 show first and second external world feature points whose correspondence relationship is determined as a result of the matching process, it is not clear which first and second external world feature points represent the same external world feature point until the result of the matching process is obtained. Therefore, in the following description of the processing procedure, the labels of the first and second external world feature points are independent. The label of the first external world feature point during the matching process is referred to as a ｈ (h=1, 2, 3...M), and the label of the second external feature point is b ｈ (h=1, 2, 3...N). The first external feature point a r1 In order to find the second external feature point corresponding to the first external feature point a r1The distance between the feature vectors of the second external feature point b is calculated. Here, the distance between the feature vectors is the norm of the difference between the two feature vectors. Then, the second external feature point b, which has the smallest inter-feature distance, is calculated. r2 the first external feature point a r1 is the corresponding second external feature point, and matching is established. However, if this smallest inter-feature distance is 0.8 times or more the second smallest inter-feature distance, it is determined that there is no corresponding second external feature point, and matching is not established. This is because the existence of another second external feature point with a close inter-feature distance may indicate an incorrect correspondence.

[0081] After the first stage of matching, a cross-check is performed. First, in the first stage of matching, the first external feature point a r1 The second external feature point b r2 Next, the second external feature point b r2 is fixed, and an external feature point is selected from the first external feature point by the above matching method. If there is no first external feature point that matches in this reverse matching, the first external feature point a r1 and the second external feature point b r2 If there is a first external feature point for which matching is established in the reverse direction, the first external feature point is set as a r3 This first external feature point a r3 is the first external feature point a r１ If it is equal to, the first external feature point a r１ and the second external feature point b r2 If the first external feature point a is not equal to the first external feature point a, the first external feature point a is considered to be equal to the first external feature point a. r１ and the second external feature point b r2 is determined to be unsuccessful. Note that the first external world feature point and the second external world feature point for which matching has been achieved may be managed by assigning the same label. In the following description, the external world feature point α, the external world feature point β, and the external world feature point γ will be used as examples of the external world feature points for which matching has been achieved in a pair of feature points in the set of first external world feature points and feature points in the set of second external world feature points.

[0082] If the number of matching external feature points is less than a predetermined number, the external areas captured by the two terminals may be different, so the video may be re-captured.

[0083] The extraction of external world feature points, the calculation of feature amounts, and the execution of matching may be performed by either terminal. The extraction of external world feature points and the calculation of feature amounts may be performed by each terminal, and the calculation results of feature amounts may be collected on one terminal to execute matching. Alternatively, external world video information may be sent from one terminal to the other, and the subsequent processing up to matching may be executed on one terminal. Furthermore, the calculation processing may be performed by server 2.

[0084] Furthermore, the measurement of the coordinate values ​​of the external world feature points may be performed for all the extracted external world feature points in the external world image, or may be limited to some of the external world feature points required for subsequent processing. When the terminal is moving, in order to obtain the coordinate values ​​of the external world feature points after the timing of capturing the external world image used for external world feature point matching, it is sufficient to capture the external world image from different capturing positions, measure the distance to the external world feature point using a moving stereo method, and obtain the coordinate values.

[0085] From among the external feature points for which matching has been established between the external image of both terminals in the processing up to this point, three external feature points that are not on the same line are selected, and a process is performed to determine the relative positional relationship between the two terminals.

[0086] FIG. 10 is a diagram showing the contents of a method (1) for acquiring the inter-terminal positional relationship of another person's information terminal with respect to a user information terminal, and is an explanatory diagram of two direction vectors defined in real space.

[0087] As shown in FIG. 10, N １ , N ２These two direction vectors can be defined, and the rotation for matching the orientations of the two terminal world coordinate systems can be calculated from the numerical representation of these two direction vectors in the terminal world coordinate systems of the in-vehicle terminal 1A and the HMD 1B. Also, the difference in the origins of the two terminal world coordinate systems can be calculated from the numerical representation of any one point in real space in the terminal world coordinate systems. Information on this rotation for matching the orientations and the difference in origins is conversion information between the terminal world coordinate systems of the in-vehicle terminal 1A and the HMD 1B. If conversion between coordinate systems can be performed, it is naturally possible to know the position of the other terminal.

[0088] In addition, N in the real space １ , N ２ If the direction vector can be measured and recognized by another method, the number of external feature points to be used can be reduced, and this method will be described later as a modified example.

[0089] FIG. 11 is a diagram showing the contents of a method (1) for acquiring the inter-terminal positional relationship of an other information terminal with respect to a user information terminal, and is a diagram showing two direction vectors in the terminal world coordinate systems of the user information terminal and the other information terminal.

[0090] Direction vector N １ , N ２ The numerical expression in the terminal world coordinate system of the vehicle-mounted terminal 1A is N （Ａ） １ , N （Ａ） ２ and the numerical expression in the terminal world coordinate system of the HMD 1B is N （Ｂ） １ , N （Ｂ） ２ These direction vectors are shown in Fig. 11. Here, the space in Fig. 11 is the coordinate space of the vehicle-mounted terminal 1A, and N （Ｂ） １ , N （Ｂ） ２ When expressed in numerical terms in the coordinate system of the vehicle-mounted terminal 1A, N （Ａ） １ , N （Ａ） ２ The direction vector is different from that of （Ｂ） １ , N （Ｂ） ２ Rotate to N （Ａ）１ , N （Ａ） ２ This explains the transformation that is superimposed on

[0091] First, N （Ａ） １ and N （Ｂ） １ Rotate around an axis perpendicular to the plane containing N （Ｂ） １ N （Ａ） １ Such a rotation Q １ is expressed by the following equation (12): １ = [1-N （Ａ） １ N （Ｂ） １ ]・・・(12)

[0092] In addition, from the beginning, N （Ｂ） １ N （Ａ） １ In the special case where １ = 1, i.e. no rotation.

[0093] After this rotation, N （Ｂ） ２ The direction vector of N （Ｂ）＃ ２ If you write this, （Ｂ）＃ ２ is expressed by the following equation (13): （Ｂ）＃ ２ =Q １ N （Ｂ） ２ Q １ ＊ ...(13)

[0094] Next, N （Ｂ）＃ ２ N （Ａ） １ Rotate around the axis and （Ａ） ２ The rotation at this time is Q ２ When written as Q ２ is expressed as the following equation (14): Q ２ = [1-[P Ｔ (N （Ａ） １ ) N （Ａ） ２ ][P Ｔ (N（Ａ） １ ) N （Ｂ）＃ ２ ]]...(14) where P Ｔ (n) is an operator that projects a vector onto a plane perpendicular to the direction vector n (defined in equation (5)). （Ｂ）＃ ２ N （Ａ） ２ In the special case where ２ = 1, i.e. no rotation.

[0095] From the above, the numerical expression of the direction vector expressed in the terminal world coordinate system of HMD1B is rotation (Q ２ Q １ ) can be converted into a numerical expression of a direction vector expressed in the terminal world coordinate system of the vehicle-mounted terminal 1A. For the following explanation, rotation Q ＢＡ is defined by the following equation (15): ＢＡ =Q ２ Q １ ...(15)

[0096] Next, the difference between the origins of the two terminal world coordinate systems is calculated. ＢＡ is expressed by the following equation (16): ＢＡ =D ＃（Ａ） -Q ＢＡ D ＃（Ｂ） Q ＢＡ ＊ ...(16)

[0097] Here, D ＃（Ａ） , D ＃（Ｂ） are numerical representations of arbitrarily selected external points in the terminal world coordinate system of the vehicle-mounted terminal 1A and the terminal world coordinate system of the HMD 1B. For example, if an external feature point α is selected and D ＃（Ａ） , D ＃（Ｂ） If you substitute specific values ​​for , ＢＡ is obtained. In this way, O ＢＡ If we use the above formula, the conversion of numerical representation between coordinate systems of any point is given by the following formula (17): （Ａ） =Q ＢＡ D （Ｂ） Q ＢＡ ＊ +O ＢＡ...(17)

[0098] The inverse transformation is given by the following equation (18): D （Ｂ） = Q ＢＡ ＊ D （Ａ） Q ＢＡ -Q ＢＡ ＊ O ＢＡ Q ＢＡ ...(18)

[0099] Here, D （Ａ） , D （Ｂ） are numerical representations of an arbitrary point in the coordinate system of the vehicle-mounted terminal 1A and the coordinate system of the HMD 1B, respectively.

[0100] The rotation Q, which is the transformation parameter between the coordinate systems of both terminals, is calculated as follows: ＢＡ and the difference between the origin position O ＢＡ By sharing this information between both terminals, each terminal can obtain the position coordinates of the other terminal in its own coordinate system and convert them into coordinates in its own coordinate system. In other words, both terminals can grasp the positional relationship between the terminals. Alternatively, one terminal may perform the coordinate conversion calculation and share the calculation results.

[0101] Note that, by using the above method, when the coordinate values ​​of the external world feature points in the terminal world coordinate system are known, it is possible to determine the position and orientation of the terminal with respect to the terminal world coordinate system using PP cameras 136A, 136B, and 136C and PP ranging sensors 137A, 137B, and 137C. Since the external world feature points can be identified using PP cameras 136A, 136B, and 136C and the coordinate values ​​of the external world feature points in the terminal local coordinate system can be measured using PP ranging sensors 137A, 137B, and 137C, it is possible to determine the transformation parameters between the terminal world coordinate system and the terminal local coordinate system using the above method.

[0102] (Variation of Method (1) for Acquiring Positional Relationship Between Terminals) If both terminals can recognize a specific direction that allows them to recognize a direction, rather than a direction between points in real space, the number of required external feature points can be reduced.

[0103] For example, the vertically upward direction can be recognized by measuring the direction of gravitational acceleration using acceleration sensors 115A and 115B on both terminals. If this vertically upward direction is considered as one of the directions in the real space described above, the number of external feature points used can be two. In other words, the direction vector from external feature point α to β is N １ and the vertical upward direction vector is N ２ If so, the transformation formula for the coordinate systems between the two terminals can be found using the same calculation procedure.

[0104] Furthermore, as another specific direction in the real space, for example, the north direction is measured by the geomagnetic sensors 117A and 117B, and a direction vector N １ In order to find the difference between the origins of both coordinate systems using equation (16), the coordinate values ​​of one point in the external world must be known in both coordinate systems, so one external feature point is used.

[0105] The specific direction in real space is not limited to the above, and other directions may be used. For example, the direction of a landmark that is sufficiently far from the terminal may be used as the direction in real space.

[0106] As described above, by using real space directions other than the directions between external field feature points, it is possible to reduce the number of external field feature points used and the calculation load for external field feature point matching.

[0107] (Method (2) of acquiring positional relationship between terminals) Next, a processing method will be described when the coordinate values ​​of external world feature points are measured only on one terminal. The other terminal measures only the direction of the external world feature points. Here, it is assumed that the coordinate values ​​of the external world feature points are measured on the vehicle-mounted terminal 1A, but it is the same even if the coordinate values ​​of the external world feature points are measured on the HMD 1B. Note that when distance measurement is performed using a moving stereo technique or when using a processing procedure that does not measure distance as in this method (2) of acquiring positional relationship between terminals, a hardware configuration that does not use distance measurement sensors 113A, 113B and PP distance measurement sensors 137A, 137B is also possible.

[0108] The vehicle-mounted terminal 1A generates external world feature point information based on coordinate value data of the external world feature points, and the HMD 1B generates external world feature point information based on directional vector data to the external world feature points. The external world feature point matching is the same as the method used in the method (1) for acquiring the positional relationship between terminals.

[0109] 12 is a diagram showing the concept of separating the positional relationship between the matched external feature points and the HMD 1B, which is the other person's information terminal, into a horizontal projection plane and a vertical projection plane. A method for determining the vertical projection plane will be described later.

[0110] First, as shown in Fig. 12, the positional relationship between the external feature points and the HMD 1B, which is the other person's information terminal, is orthogonally projected onto a horizontal projection plane and a vertical projection plane, and the positional relationship within the horizontal plane and the positional relationship within the vertical plane are considered separately. The vertical direction required for projection onto the horizontal projection plane can be recognized by both the in-vehicle terminal 1A and the HMD 1B, and is performed by measuring the direction of gravitational acceleration using acceleration sensors 115A and 115B, for example.

[0111] 12, the terminal position X of the HMD 1B and the positions of the external world feature points α, β, and γ in the real space projected onto a horizontal projection plane are represented by terminal position X' and external world feature points α', β', and γ'. Also, the terminal position X of the HMD 1B and the positions of the external world feature points α, β, and γ projected onto a vertical projection plane are represented by terminal position X'' and external world feature points α'', β'', and γ''.

[0112] When a taxi approaches a passenger, the difference in vertical position between the taxi and the passenger is usually not large enough to cause inconvenience during the meeting. Therefore, if the relative positional relationship between the two in the horizontal projection plane is known, support for the meeting place can be provided, but the processing for a general case will also be explained. First, the processing content of the in-vehicle terminal 1A estimating the terminal position X' of the other person's information terminal in the horizontal projection plane will be explained.

[0113] FIG. 13 is a diagram showing the contents of a method (2) for acquiring the inter-terminal positional relationship between a user information terminal and another person's information terminal.

[0114] The coordinate values ​​of the external feature points α, β, and γ in the terminal world coordinate system of the vehicle-mounted terminal 1A are measured by the vehicle-mounted terminal 1A.

[0115] On the other hand, in the HMD 1B, the direction vectors from the terminal to the external feature points α, β, and γ are expressed numerically in the coordinate system of the HMD 1B, N α , N β , N γ Ask for.

[0116] Direction vector N to the external feature point α , N β , N γ Once this is determined, the vertical upward direction vector is N Ｖ and the angle of view (θ') between the external feature points as seen from the position of the HMD 1B (terminal position X') in the horizontal projection plane. αβ , θ' βγ , θ' γα ) is calculated from the dot product between each direction vector. Note that -SC(ab) represents the dot product of vector a and vector b, and each angle of view can be calculated using the following formula: θ' αβ = cos －１ (-Sc([P Ｔ (N Ｖ ) N α ][P Ｔ (N Ｖ ) N β ]))...(19) θ' βγ = cos －１ (-Sc([P Ｔ (N Ｖ ) N β ][P Ｔ (N Ｖ ) N γ ]))...(20) θ' γα = cos －１ (-Sc([P Ｔ (N Ｖ ) N γ ][P Ｔ (N Ｖ ) N α ])) ... (21)

[0117] According to the theorem of circular angles, the angle between the external feature point α' and the external feature point β' is θ' αβThe set of points where X' is the terminal position is a point somewhere on the circumference of this circle. The same can be said for other sets of feature points. Therefore, three circles are created in total, and the location of terminal position X' in the horizontal projection plane of HMD 1B can be determined as the intersection of two of these circles (Figure 13).

[0118] To determine the specific coordinate value of X', select a horizontal projection plane, which passes through β and is Ｖ A plane perpendicular to β is selected. In other words, β' is equal to β. A two-dimensional coordinate system is defined on this plane. Hereafter, it is called coordinate system A1. The origin is β. A direction vector N from β to α' is Ｔ１１ is one of the coordinate axes. Direction vector N Ｔ１１ is given by the following formula: N Ｔ１１ = [P Ｔ (N Ｖ )(α-β)]...(22)

[0119] And the direction vector N Ｔ１１ , N Ｖ A direction vector N rotated 90 degrees around the axis Ｔ１２ Let N be the other coordinate axis. Ｔ１２ is given by the following formula: N Ｔ１２ = H(N Ｖ ) N Ｔ１１ H (N Ｖ ) ＊ ...(23)

[0120] Here, H(n) is a 90 degree rotation around the axis of the direction vector n (defined in equation (3)).

[0121] The numerical expression in the above coordinate system A1 is represented by a superscript (A1). The numerical expression of an arbitrary point D in the coordinate system A1 can be expressed in component notation as follows: D （Ａ１） = (-Sc(N Ｔ１１ P Ｔ (N Ｖ )(D-β)), -Sc(N Ｔ１２ P Ｔ (N Ｖ )(D-β)))...(24)

[0122] α' （Ａ１） , β' （Ａ１） , γ'（Ａ１） The coordinate value and the angle of view (θ' αβ , θ' βγ , θ' γα ) has been found, the coordinate value X' is calculated using the method for finding X' from the intersection of the circles described above. （Ａ１） The coordinate value X' thus obtained is （Ａ１） The component expression of X' is as follows: （Ａ１） = ((X' （Ａ１） ) １ , (X' （Ａ１） ) ２ ) ... (25)

[0123] Similarly, if the terminal position X'' of the HMD 1B in the vertical projection plane is found, the position of the vehicle-mounted terminal 1A of the HMD 1B in the terminal world coordinate system can be determined. Here, the horizontal direction vector N Ｈ It is necessary to determine the horizontal vector N Ｈ In this case, the vertically upward direction vector N Ｖ is a direction vector rotated 90 degrees around the axis of the direction vector from X' to β, and N Ｈ Define N Ｈ The numerical expression in each coordinate system is as follows: Ｈ （Ａ） = H([β （Ａ） -X' （Ａ） ]) Nv （Ａ） H ([β （Ａ） -X' （Ａ） ]) ＊ ... (26) N Ｈ （Ｂ） = H([P Ｔ (N Ｖ （Ｂ） ) N β （Ｂ） ]) Nv （Ｂ） H ([P Ｔ (N Ｖ （Ｂ） ) N β （Ｂ） ]) ＊ ...(27)

[0124] Here, [β' in formula (26) （Ａ） -X'（Ａ） ] is represented by the following formula, and X' （Ａ１） can be determined if [β （Ａ） -X' （Ａ） ]=-[(X' （Ａ１） ) １ N Ｔ１１ （Ａ） +(X' （Ａ１） ) ２ N Ｔ１２ （Ａ） ]・・・(28)

[0125] This N Ｈ By projecting each position onto the vertical projection plane using the above equation and performing the same processing as that performed within the horizontal projection plane, the position of the terminal position X'' within the vertical projection plane of the HMD 1B can be determined.

[0126] First, similarly to the case in the horizontal projection plane (FIG. 13), the angle of view (θ'') between the external feature points as seen from the position of the HMD 1B (terminal position X'') in the vertical projection plane is calculated. αβ , θ′′ βγ , θ′′ γα ) can be calculated using the following formula: θ'' αβ = cos －１ (-Sc([P Ｔ (N Ｈ ) N α ][P Ｔ (N Ｈ ) N β ]))...(29) θ'' βγ = cos －１ (-Sc([P Ｔ (N Ｈ ) N β ][P Ｔ (N Ｈ ) N γ ]))...(30) θ'' γα = cos －１ (-Sc([P Ｔ (N Ｈ ) N γ ][P Ｔ (N Ｈ ) N α ])) ... (31)

[0127] Then, similarly to the processing within the horizontal projection plane, the position of the terminal position X'' within the vertical projection plane of the HMD 1B can be determined as the intersection of the circles.

[0128] To find the specific coordinate value of X'', select one vertical projection plane. Here, the plane passing through β is N Ｈ Select a plane that is perpendicular to . In other words, β'' is equal to β. The vertical projection plane selected here is a plane obtained by rotating the vertical projection plane selected above by 90 degrees around the axis of the direction vector from X' to β. In other words, X' is located within this vertical projection plane.

[0129] Now, let us define a two-dimensional coordinate system on this plane. Hereafter, we will call it coordinate system A2. The origin is β. The direction vector N from β to α'' is Ｔ２１ is one of the coordinate axes. Direction vector N Ｔ２１ is given by the following formula: N Ｔ２１ = [P Ｔ (N Ｈ )(α-β)]...(32)

[0130] And the direction vector N Ｔ２１ , N Ｈ A direction vector N rotated 90 degrees around the axis Ｔ２２ Let N be the other coordinate axis. Ｔ２２ is given by the following formula: N Ｔ２２ = H(N Ｈ ) N Ｔ２１ H (N Ｈ ) ＊ ...(33)

[0131] The numerical expression in the above coordinate system A2 is expressed by a superscript (A2). The numerical expression of an arbitrary point D in the coordinate system A2 can be expressed in component notation as follows: D （Ａ２） = (-Sc(N Ｔ２１ P Ｔ (N Ｈ )(D-β)), -Sc(N Ｔ２２ P Ｔ (N Ｈ )(D-β)))...(34)

[0132] α'' （Ａ２） , β'' （Ａ２） , γ′′ （Ａ２） The coordinate value and the angle of view (θ'' αβ , θ′′ βγ , θ′′ γα) has been found, the coordinate value X'' can be calculated by the same method as in Figure 13 to find X'' from the intersection of the circle. （Ａ２） The coordinate value X'' obtained in this way is （Ａ２） The component expression of X'' is as follows: （Ａ２） = ((X'' （Ａ２） ) １ , (X'' （Ａ２） ) ２ ) ... (35)

[0133] From the above results, the numerical expression X in the vehicle-mounted terminal 1A of the terminal position X of the HMD 1B is （Ａ） Here, it should be noted that X' is a point included in both the horizontal and vertical projection planes, depending on how the horizontal and vertical projection planes are selected (Fig. 14). Therefore, the positional relationship between X' and X is N Ｖ If we pay attention to the alignment of the points in the direction, X is a point in the selected vertical projection plane. Therefore, the position of point X'' projected from X onto the vertical projection plane is equal to X. Therefore, X （Ａ） is calculated using the following formula: X （Ａ） =X'' （Ａ） = (X'' （Ａ２） ) １ N Ｔ２１ （Ａ） + (X'' （Ａ２） ) ２ N Ｔ２２ （Ａ） +β （Ａ） ...(36)

[0134] Here, we will explain points to note when estimating the terminal position X of the HMD 1B from the in-vehicle terminal 1A using the theorem of circular angles shown in FIG. 13 . As shown in FIG. 15 , there are cases where the three external field feature points α', β', and γ' and the terminal position X' in the horizontal projection plane of the HMD 1B are located on the same circumference. In this case, the terminal position X' of the HMD 1B cannot be determined using the process shown in FIG. 13 . In this case, another external field feature point that is not located on the same circumference is substituted (substitute external field feature point). The substitution with the substitute external field feature point is achieved by the processor 125A of the in-vehicle terminal 1A automatically setting a fourth point when the in-vehicle terminal 1A executes the process of FIG. 13 using the three external field feature points α', β', and γ' but is unable to estimate the terminal position X'. The same applies to processing in the vertical projection plane. To prepare for such cases, it is desirable to set the number of external field feature points whose positions are to be measured to four or more.

[0135] Below, we will determine the three-dimensional positional relationship between the user information terminal and the other person's information terminal, but in a normal meeting situation, the user information terminal and the other person's information terminal are located approximately on the same plane, so it is sufficient to only determine the positional relationship within the horizontal projection plane.

[0136] Through the processing up to this point, the terminal position X of the HMD 1B and the positions of each external feature point in the coordinate system of the vehicle-mounted terminal 1A are known. Now, if any two points are selected from the external feature points, for example, the external feature point β and the external feature point γ, a direction vector N from the terminal position X to the external feature point β is obtained. １ and a direction vector N from the terminal position X to the external feature point γ. ２ The numerical expression in the coordinate system of the vehicle-mounted terminal 1A is obtained (FIG. 16).

[0137] On the other hand, in the HMD 1B, the position and orientation sensor 135B of the HMD 1B detects the coordinate value of the HMD 1B in the coordinate system of the HMD 1B (i.e., the coordinate value of the terminal position X) and the direction vector N １ and N ２ We know the numerical representation of N １ and N ２ It is assumed that external feature points are selected in advance so that they do not have the same direction.

[0138] Now, the direction vector N in two different external worlds １ and N ２ Since the numerical expressions in the respective terminal world coordinate systems are obtained, the rotation Q is used to convert the numerical expression of the direction vector expressed in the coordinate system of the HMD 1B into the numerical expression of the direction vector expressed in the coordinate system of the vehicle-mounted terminal 1A by the method described in the method (1) for acquiring the positional relationship between the terminals. ＢＡ can be obtained (FIGS. 10 and 11).

[0139] Next, similarly to equation (16), the difference between the coordinate system origins is calculated using information on points whose numerical expressions in the coordinate systems of both terminals are known. ＢＡ is as follows: O ＢＡ =X （Ａ） -Q ＢＡ X （Ｂ） Q ＢＡ ＊ ...(37)

[0140] Here, X （Ａ） , X （Ｂ） are numerical representations of the terminal position X in the coordinate system of the vehicle-mounted terminal 1A and the coordinate system of the HMD 1B, respectively.

[0141] As with the method (1) for obtaining the positional relationship between terminals, the rotation Q ＢＡ and the difference between the origin position O ＢＡ By sharing this information between both terminals, each terminal can obtain the position coordinates of the other terminal in its own coordinate system and convert them into coordinates in its own coordinate system. In other words, both terminals can grasp the positional relationship between the terminals. Alternatively, one terminal may perform the coordinate conversion calculation and share the calculation results.

[0142] (Modification of Method (2) for Acquiring Positional Relationship Between Terminals) In the above, the projection plane is set based on the vertical direction that can be recognized by both terminals, but this is not limited to this as long as it is an external direction that can be recognized by both terminals. For example, the direction of north may be measured by the geomagnetic sensors 117A and 117B and used as the reference direction. Also, for example, the direction of a landmark that is sufficiently far from the terminals may be used as the reference direction. This reference direction vector is referred to as "N" in the above processing.Ｖ " and the same procedure can be performed.

[0143] (Summary of methods (1) and (2) for acquiring the positional relationship between terminals) Here we will summarize the methods for acquiring the positional relationship between two terminals. Hereinafter, the entity that processes data is a processor installed on one of the terminals or the server. The data to be processed is sent from each terminal to the terminal or server that will perform the processing, and the terminal or server that performed the data processing sends the processing results to the terminal or server that will use the processing results. Ultimately, each terminal uses the data processing results to recognize the relative position of the other terminal with respect to its own terminal, and uses this as support information for executing the rendezvous.

[0144] (1) The approximate location of the meeting is recognized by both terminals. This recognition method may involve, for example, sharing the name of the meeting place. The meeting place may be specified by the name of a street, the name of a nearby building or facility, or an address. Alternatively, location recognition information obtained by a common positioning system implemented on both terminals may be used. This common positioning system may be, for example, a positioning method that uses satellites such as GPS. Note that in the present invention, even if the specific positioning methods are different, those that use common location expressions such as latitude, longitude, and altitude are considered to be the same common positioning system.

[0145] (2) Both devices capture images of the outside world near the meeting place (first and second outside images). Because the same area of ​​the outside world needs to be captured, information about the object to be captured may be shared between both devices. The information about the object to be captured may include, for example, the name of the building or facility to be captured, the position of the object to be captured based on a common positioning system, or information about the location and direction of the image to be captured based on a common positioning system.

[0146] (3) Either one of the terminals or the server compares the images captured by both terminals and performs matching of external feature points. Distributed processing is also acceptable.

[0147] (4) In either the terminal or the server, for a pair of matching external feature points in the set of first external feature points and external feature points in the set of second external feature points, information on the relative positional relationship between the external feature points and the terminal in the positioning system of each terminal is obtained. Here, "information on the relative positional relationship between the external feature points and the terminal" refers to "information on the coordinate values ​​of the terminal position and the coordinate values ​​of the external feature points" or "information on the coordinate values ​​of the terminal position and the direction of the external feature points as seen from the terminal position." For at least one of the terminals, "information on the relative positional relationship between the external feature points and the terminal" needs to be "information on the coordinate values ​​of the terminal position and the coordinate values ​​of the external feature points." Here, "the positioning system of each terminal" refers to a positioning system by which each terminal recognizes its own position and orientation, and more specifically, refers to a measurement method and expression format for recognizing coordinate values ​​and direction vectors in a terminal world coordinate system defined by each terminal.

[0148] The direction of the external feature point as seen from the terminal position in the positioning system of each terminal described above can be calculated from the external world image captured by that terminal, as shown in Fig. 8. The calculation itself can be performed by other terminals or servers if the terminal position coordinates and image angle of view information are transmitted together with the external world image information.

[0149] In order to obtain the coordinate values ​​of the external feature points in the positioning system of each terminal described above, it is necessary to measure the distance from the terminal position to the external feature point in addition to information on the direction of the external feature point as seen from the terminal position. Distance measurement methods are broadly divided into methods using ranging sensors 113A, 113B, and 113C, and a moving stereo method using images captured at different shooting locations. While methods using ranging sensors 113A, 113B, and 113C are performed at the terminal, the moving stereo method may transmit image information from different shooting locations together with information on the shooting locations to a partner terminal (another person's information terminal as seen from the user information terminal, or the user information terminal as seen from the other person's information terminal) or a server, and the distance may be calculated at the partner terminal or server. Hereinafter, in distance measurement using the moving stereo method, it is assumed that the executing entity holds the information on the shooting location and image information required for measurement, or acquires that information from the shooting terminal. Furthermore, it is necessary to match external feature points in the images between the multiple images used in the moving stereo method, and this matching can be performed using the same method as the external feature point matching between the images of both terminals.

[0150] The distance between the device and the external world feature point may be measured before performing external world feature point matching. In this case, the device that performed the distance measurement transmits list information that pairs the position information of the external world feature in the external world image with the distance information to the device or server that uses the distance information. Here, the execution of distance measurement may be limited to the external world feature points extracted as external world feature points, or distance measurement to measurable external world points may be performed. In this method of measuring distance before external world feature point matching, there is a possibility that distance information is not available for the matched external world feature point. In this case, the matching may be treated as not being successful, or, as described below, distance measurement of the relevant external world feature point may be performed after external world feature point matching.

[0151] Measurement of the distance between the terminal and the external feature points may be performed after performing external feature point matching. In this case, if the terminal or server performing the distance measurement is not performing the matching process, the terminal or server that performed the matching process transmits information about the external feature points for which matching was achieved to the terminal or server performing the distance measurement. After the distance measurement is performed, the entity performing the distance measurement transmits the distance information to the terminal or server that requires it.

[0152] (5) Using the method (1) or (2) for acquiring the positional relationship between terminals, the terminal or server determines the "rotation and origin difference," which is the coordinate system transformation parameter between the world coordinate systems of both terminals. After the coordinate system transformation parameter is determined, the executing entity for calculating the coordinate system transformation parameter transmits the information to the terminal or server that uses the coordinate system transformation parameter.

[0153] Here, we summarize how to calculate coordinate system transformation parameters in methods (1) and (2) for acquiring the inter-device positional relationship, including both methods. First, a rotation parameter for aligning the orientations of the two device world coordinate systems (first device world coordinate system and second device world coordinate system) is calculated from the direction vector values ​​of two direction vectors defined in the external world in two device world coordinate systems. Next, the difference between the origins of the two device world coordinate systems is calculated from the coordinate values ​​of one position defined in the external world in the two device world coordinate systems. Here, the two direction vectors defined in the external world refer to two direction vectors selected from the direction vector from the device position to the external world feature point position or the direction vector from one external world feature point to another external world feature point. Furthermore, the one position defined in the external world refers to either one of the device positions or one external world feature point position.

[0154] (6) Each terminal transmits its own location information to the other terminal or server that uses this location information in advance or each time it is updated. This location information may be a numerical representation of the terminal's positioning system, or may be converted into a numerical representation of the other terminal's positioning system after the coordinate system transformation parameters are calculated. Alternatively, both the numerical representations of the positioning systems of both terminals may be transmitted. Each terminal or server executes a process to support rendezvous based on the terminal's location information. Examples of rendezvous support processes include superimposing the location of the other terminal on an external image, or using the information to control the driving of a vehicle such as a car to guide the other terminal to its location.

[0155] (Modification (1) common to methods (1) and (2) for acquiring positional relationship between terminals) In the method of the present invention, a direction vector to an external feature point is acquired from an external image captured by the terminal, so it is preferable that distortion of the optical system is corrected and distortion of the captured image is minimized. Therefore, if the outer cameras 111A, 111B, and 111C of the terminal have a low-distortion shooting mode, the external image for acquiring the relative position may be automatically captured in the low-distortion shooting mode. Figure 17 shows an example in which a message for switching to low-distortion shooting mode is superimposed on the captured image preview screen. Switching to low-distortion shooting mode can improve the accuracy of estimating the relative position.

[0156] (Variation (2) common to methods (1) and (2) for acquiring positional relationship between terminals) In the method of the present invention, it is essential that both terminals capture images of the same area of ​​the outside world, so the image used to acquire the relative position may be wide-angle image. Therefore, if the outer cameras 111A, 111B, and 111C of the terminals have a wide-angle mode, wide-angle image may be automatically captured when capturing image for acquiring the relative position. Figure 18 shows an example in which a message for switching to wide-angle capture mode is superimposed on the captured image preview screen. Switching to wide-angle capture mode increases the certainty of capturing images of the same area of ​​the outside world, reducing the risk of taking additional images.

[0157] (Modification (3) common to methods (1) and (2) for acquiring the positional relationship between terminals) As with modification (2) described above, a shooting guide may be displayed as a method for increasing the certainty that both terminals will shoot the same area of ​​the outside world. The shooting guide here is to display the image shot by the other terminal on the shot image preview screen of the terminal itself when the other terminal shoots an image of the outside world first. An example of such a display is shown in FIG. 19. By displaying the image shot by the other terminal and providing the user with guidance on the shooting direction, the certainty that the same area of ​​the outside world will be shot increases, and the risk of taking additional images can be reduced.

[0158] (Modification (4) common to methods (1) and (2) for acquiring positional relationship between terminals) In addition to the above-described modification (3), another shooting guide may be displayed as a method for increasing the certainty that the same area of ​​the outside world is photographed on both terminals. The shooting guide here is to display an AR object or the like indicating the target shooting point on the shooting video preview screen of the terminal itself, based on target shooting point information based on a common positioning system such as GPS that allows both terminals to share position recognition. An example of such a display is shown in FIG. 20. By displaying an AR object or the like indicating the target shooting point on the shooting video preview screen of the terminal itself, the certainty that the same area of ​​the outside world is photographed is increased, and the risk of taking additional photographs can be reduced.

[0159] (Modification (5) common to methods (1) and (2) for acquiring positional relationship between terminals) In the above process, three matched external feature points are used (one or two external feature points are used when a specific direction in real space is used) to calculate the rotation and origin difference, which are the transformation parameters of the coordinate systems of both terminals. However, more external feature points may be used. That is, multiple coordinate system transformation parameters may be obtained from multiple sets of three external feature points, and their average value may be used. When a specific direction in real space is used, the number of external feature points is determined to be the number required for that case. In this case, when looking at the distribution of the coordinate system transformation parameters, any data that deviates significantly from the average value may be considered anomalous data and excluded from the data for calculating the average value, and the calculation may be repeated. Well-known methods such as the Smirnoff-Grubbs test may be used to determine anomalous data. For example, if the standard deviation of the distribution is σ, data that deviates by 3σ or more from the average value may be determined to be anomalous data.

[0160] The process of calculating the average value may be performed on the coordinate values ​​in the terminal's own world coordinate system of the location of the other terminal estimated using the coordinate system transformation parameters. In this case, when examining the distribution of estimated coordinate values, any values ​​that deviate significantly from the average value may be regarded as abnormal data and excluded from the data for calculating the average value, and the calculation may be retried. The coordinate system transformation parameters corresponding to these abnormal estimated coordinate values ​​are also recognized as abnormal data.

[0161] Among the three external field feature points corresponding to coordinate system transformation parameters determined as abnormal data using the above-described method, there is a possibility that at least one is a false external field feature point that has been accidentally matched between two images. Therefore, external field feature points belonging to a triplet of external field feature points corresponding to abnormal data equal to or greater than the set number of pairs may be determined as false external field feature points, and the matching may not be established. In other words, this is additional screening of external field feature point matching based on the calculation results of coordinate system transformation parameters.

[0162] After the above-described additional screening of external boundary feature points is performed, a plurality of coordinate system transformation parameters may be newly calculated from a plurality of triplet sets of external boundary feature points, and the process of calculating the average value thereof may be repeated.

[0163] By performing the processing of this modified example, it is possible to improve the accuracy of estimating the relative position.

[0164] (Variation (6) Common to Methods (1) and (2) for Acquiring Positional Relationship Between Terminals) If the number of matched external world feature points is less than a set number, or if the variation in the transformation parameters of the coordinate systems of both terminals calculated using the matched external world feature points is greater than a set amount, it is possible that the locations captured by both terminals are different, or that the captured ranges of the images have little overlap. In this case, if additional capture is possible, a message urging the users of both terminals to take additional capture may be displayed by visual display or audio notification. This increases the likelihood that the same area of ​​the external world will be captured if additional capture is performed ( FIG. 21A ). Furthermore, if it is determined that a terminal has moved away from the capture location, the terminal may stop issuing the above-mentioned message and notify the other terminal that it is unable to take additional capture, prompting the user of the other terminal to take additional capture ( FIG. 21B ). This improves the effectiveness of the message.

[0165] Furthermore, when the moving terminal (vehicle-mounted terminal 1A) approaches the meeting place, only the vehicle-mounted terminal 1A may additionally capture an external image of the vicinity of the meeting place. This changes the image on the vehicle-mounted terminal 1A side to an image captured from a location closer to the location captured by the HMD 1B. This increases the reliability of matching of external world feature points, increases the number of matching external world feature points, and improves the accuracy of estimating the relative positions of the two terminals.

[0166] (Information Display) FIG. 22 is a diagram showing an example of information display on the user information terminal (vehicle-mounted terminal 1A) based on terminal position estimation for meeting support.

[0167] 22 , an outside image 1200 captured by the outer camera 111A is displayed on the display 119A of the in-vehicle terminal 1A. Furthermore, virtual objects 1210 and 1211 indicating the terminal position X of the HMD 1B calculated by the processor 125A of the in-vehicle terminal 1A are displayed superimposed on the outside image 1200 of the display 119A. The terminal position X corresponds to the estimated position of the passenger.

[0168] FIG. 23 is a diagram showing an example of information display on the other person's information terminal (HMD 1B) based on terminal position estimation for meeting support.

[0169] 23 , an outside world image 1300 captured by the outer camera 111B is displayed on the display 119B of the HMD 1B. Furthermore, the HMD 1B receives information indicating the relative positional relationship between the in-vehicle terminal 1A and the terminal position X of the HMD 1B, calculated by the processor 125A of the in-vehicle terminal 1A. The processor 125B of the HMD 1B then calculates the terminal position of the in-vehicle terminal 1A relative to the HMD 1B using the information indicating the relative positional relationship, and displays virtual objects 1310 and 1311 indicating the position of the in-vehicle terminal 1A superimposed on the outside world image 1300 on the display 119B. The objects 1310 and 1311 are virtual objects indicating the estimated position of a taxi.

[0170] This embodiment makes it possible to provide highly accurate rendezvous guidance.

[0171] The information display in the automobile using the in-vehicle terminal 1A may be a HUD (Head Up Display).

[0172] If the estimated location of the other party's terminal is not included in the range shown on the display, the direction to be displayed may be guided by an arrow display, audio, etc. The direction of the meeting party may be presented by sound image localization using 3D audio.

[0173] Furthermore, the information on the location of the other party's terminal may be used not only to confirm the other party but also to control the speed of the car so that the car stops quickly and without abrupt stops in front of the other party. The speed control may be used to recommend a speed to the driver, or for speed control in a driving support system or an automatic driving system.

[0174] 24 is a flowchart showing the processing flow of the meeting support system 100 according to this embodiment. Although the server 2 is not explicitly shown, some of the processing may be performed by the server 2 to distribute the processing load.

[0175] First, the in-vehicle terminal 1A and the HMD 1B accept the setting of a meeting place by the user and start processing. A specific example of use is when a user of the HMD 1B requests a taxi to pick them up and sets the meeting place on the in-vehicle terminal 1A installed in the requested taxi. After setting the meeting place, the in-vehicle terminal 1A and the HMD 1B, for example, capture external world images near the meeting place using the outer cameras 111A and 111B, acquire terminal position coordinates at the time of capture in the respective terminal world coordinate systems, and measure the distance from the terminal position to an external world feature point (S01A, S01B). The distance measurement may be performed using distance measurement sensors 113A and 113B such as LiDAR or TOF, or a moving stereo method using multiple images captured at different positions. Note that if the method (2) for acquiring the positional relationship between terminals is used, distance measurement can be omitted in one of the terminals, so distance measurement is not performed in the terminal that omits distance measurement. Alternatively, distance measurement at this stage may not be necessary in the terminal that measures distances to external feature points after feature point matching.

[0176] Next, the in-vehicle terminal 1A and the HMD 1B share the captured external world images via the network communication devices 120A and 120B (S02A, S02B). Hereinafter, the transmission and sharing of information will be assumed to be performed via the network communication devices 120A, 120B, and 220, and an explicit description thereof will be omitted. Here, the external world image data may be aggregated only in the terminal that performs external world feature point matching or in the server 2.

[0177] Next, the processors 125A and 125B of the in-vehicle terminal 1A and the HMD 1B compare the images captured by both terminals, match the external feature points, and share the matching results (S03A, S03B). Here, the matching process may be executed by the processor of either terminal, or by the processor 225 of the server 2. Hereinafter, in text expressions in which the in-vehicle terminal 1A, the HMD 1B, or the server 2 is the processing subject, it is assumed that the processing subjects at the hardware component level shown in Figures 2 and 3 are the processors 125A, 125B, and 225.

[0178] Here, the vehicle-mounted terminal 1A and the HMD 1B determine whether the number of matching external feature points is equal to or greater than a predetermined number (S04A, S04B).

[0179] If the determinations in S04A and S04B are negative (S04A, S04B: No), the in-vehicle terminal 1A and the HMD 1B return to steps S01A and S01B. If one of the terminals, for example, the HMD 1B, has moved from the shooting point to a distant location and is unable to take additional images, the HMD 1B skips step S01B and repeats the process from S02B using the previous image.

[0180] If the determinations in S04A and S04B are positive (S04A and S04B: Yes), the in-vehicle terminal 1A and the HMD 1B perform distance measurement for the matched external field feature points (S05A and S05B). If distance measurement has been performed in the steps of S01A and S01B, distance measurement at this step may be omitted. Furthermore, distance measurement is not performed in terminals that omit distance measurement.

[0181] In the distance measurement in steps S05A and S05B, if the terminal has not moved from the shooting point of the video used for matching, distance measurement is possible using distance measurement sensors 113A and 113B such as LiDAR or TOF. If the terminal moves and distance measurement of the external world feature point by distance measurement sensors 113A and 113B becomes impossible, the in-vehicle terminal 1A and HMD 1B capture multiple images in advance with the outer cameras 111A and 111B by shifting the shooting point within a range where the same external world area can be captured, and measure the distance from the shooting point to the external world feature point by the moving stereo method.

[0182] Next, the in-vehicle terminal 1A and the HMD 1B calculate the coordinates of the external world feature points in their respective terminal world coordinate systems using the results of the distance measurements from the terminal position to the external world feature points performed up to this stage, and share the terminal position coordinates and the external world feature point coordinates (S06A, S06B). Note that the calculation of the external world feature point coordinates can be performed after the distance measurement, so it may be performed before this stage. Furthermore, the in-vehicle terminal 1A and the HMD 1B may transmit the results of the distance measurement to another terminal or the server 2 to have the external world feature point coordinates calculated, and share the results.

[0183] Next, the in-vehicle terminal 1A and the HMD 1B use the terminal position coordinates and the external world feature point coordinates to calculate and share transformation parameters between the world coordinate systems of both terminals (S07A, S07B). Note that the in-vehicle terminal 1A and the HMD 1B may transmit information on the terminal position coordinates and the external world feature point coordinates to a terminal or server 2 that performs calculation processing of the terminal world coordinate system transformation parameters, have the transformation parameters between the world coordinate systems calculated, and share the results. The process of having the other party's terminal or server 2 perform the calculation may be performed consecutively from the stages of steps S06A and S06B.

[0184] Next, the in-vehicle terminal 1A and the HMD 1B transmit their own current position information to each other terminal, and use the conversion parameters between the world coordinate systems to grasp the position of the other terminal in the world coordinate system (S08A, S08B). This calculation of the positions of each terminal may also be performed in one of the terminals or in the server 2.

[0185] At this point, the vehicle-mounted terminal 1A and the HMD 1B determine whether to update the external world image (S09A, S09B). This determination is made based on whether the variation in the estimation result of the partner terminal position exceeds a certain limit and on how close the moving terminal (vehicle-mounted terminal 1A) has come to the meeting place.

[0186] The variance of the estimation results is determined by, for example, updating the external world image when the standard deviation of the estimated position distribution is equal to or greater than a predetermined value. The estimated positions are calculated as three-dimensional or two-dimensional values, and the standard deviation mentioned above refers to the standard deviation of the distribution of distances of each estimated position from the average position.

[0187] The degree of approach of the vehicle-mounted terminal 1A to the meeting place is determined by the distance between the terminal position of the vehicle-mounted terminal 1A and the estimated terminal position of the HMD 1B. The image is updated when a predetermined threshold value for the estimated inter-terminal distance is reached. Multiple threshold values ​​may be defined. A standard deviation of the target estimated position is determined for each threshold value, and it is determined that the image should be updated if the estimated position is not equal to or less than that standard deviation.

[0188] If the determinations in S09A and S09B are positive (S09A, S09B: Yes), the in-vehicle terminal 1A and the HMD 1B return to the processing of steps S01A and S01B. However, once the determinations in S09A and S09B become negative (S09A, S09B: No) and the output of meeting support information (S10A, S10B) has started, the in-vehicle terminal 1A and the HMD 1B continue the processing of steps S08A, S08B, S10A, and S10B in parallel using the existing coordinate transformation parameters until the external world image is updated and new coordinate transformation parameters are obtained according to the flow after returning to the processing of steps S01A and S01B.

[0189] If the determinations in S09A and S09B are negative (S09A, S09B: No), the in-vehicle terminal 1A and the HMD 1B output meeting support information (S10A, S10B). Note that if the processing of steps S08A, S08B, S10A, and S10B has been continued using existing coordinate transformation parameters, the in-vehicle terminal 1A and the HMD 1B change to newly obtained coordinate transformation parameters and continue the processing of steps S08A, S08B, S10A, and S10B until the termination condition (S11A, S11B) is satisfied.

[0190] In steps S10A and S10B, the in-vehicle terminal 1A and the HMD 1B each output information for meeting support based on the estimated position of the other terminal. Note that in the in-vehicle terminal 1A, the information output for meeting support may be for automatic driving control. In addition, in the HMD 1B, if the user simply waits at the meeting place, there is no need to use information for meeting support, such as displaying the estimated position of the other terminal.

[0191] Next, if the termination condition is satisfied (S11A, S11B: Yes), the in-vehicle terminal 1A and the HMD 1B terminate the process, and if the termination condition is not satisfied (S11A, S11B: No), the process returns to steps S01A and S01B. The termination condition for the in-vehicle terminal 1A may be determined to be satisfied, for example, when the in-vehicle terminal 1A arrives at the meeting place, picks up a passenger, and turns on the taxi meter. The termination condition for the HMD 1B may be determined to be satisfied when the distance between the terminal positions of the in-vehicle terminal 1A and the HMD 1B is less than a distance threshold, for example, 1 meter.

[0192] In this embodiment, both devices capture images of the external world around the meeting place at the time of the meeting, and the location of the other device in its own device's world coordinate system is estimated by comparing the numerical representations of external feature points in the images in each device's world coordinate system. This makes it possible to easily and accurately estimate the location of the other device without being affected by position measurement errors in a general-purpose common positioning system, such as reflection of satellite radio waves by buildings, and without the need to prepare a huge amount of external world images of the area around potential meeting places in advance. Furthermore, compared to using library images, this invention uses images from the time of the meeting, which has the advantage of not being affected by changes in the image that may occur between the time the library images were captured and the time of use.

[0193] <Second Embodiment> In a second embodiment, in order to perform more precise speed control of the vehicle, some or all of the external field feature points may be set on the body of the vehicle on which the in-vehicle terminal 1A is mounted. Fig. 25 is a diagram showing an example of setting the external field feature points in the second embodiment. In this case, the image of the vehicle used for matching the external field feature points is one that has been captured in advance. If it is possible to identify which part of the vehicle body is involved, the coordinate values ​​in the terminal local coordinate system can be determined, and by referring to the time of capturing the image on the HMD 1B, the coordinate values ​​in the terminal world coordinate system can be calculated.

[0194] 25, in a situation where a car is approaching the vicinity of the meeting point, the external world feature points set on the car body will be closer to the HMD 1B than the external world feature points set on a building, etc., and furthermore, for the in-vehicle terminal 1A, these are external world feature points whose position coordinates are known accurately, so the accuracy and precision of position estimation will be improved more by setting external world feature points on the car body than by setting external world feature points on a building, etc. In this case, in order to confirm that the car is the one to be met, an identification number or the like displayed on the car body that can be seen in the video may be recognized.

[0195] <Third embodiment> In a third embodiment, the information display (FIG. 22) of the in-vehicle terminal 1A may also be displayed on the HMD 1B, and the passenger may check whether the passenger has been correctly identified by estimating the position of the destination point. If the passenger has been incorrectly identified, the HMD 1B may transmit a message to that effect to the in-vehicle terminal 1A. The passenger may manually input the determination as to whether the passenger has been correctly identified into the HMD 1B, or the processor 125B of the HMD 1B may automatically determine whether the passenger has been correctly identified.

[0196] Furthermore, if the passenger is incorrectly captured in the image sent from the vehicle-mounted terminal 1A, the true passenger position can be indicated to the vehicle-mounted terminal 1A based on the position in the image, thereby correcting the passenger's identification. If the passenger is not captured in the image sent from the vehicle-mounted terminal 1A, the HMD 1B may notify the vehicle-mounted terminal 1A of the direction in which the true passenger position deviates from the image. Furthermore, if the estimation of the passenger position is incorrect, both terminals may re-capture the external world image around the target point, and the relative position may be re-estimated.

[0197] After the passenger to be met is confirmed on the video, the speed of the vehicle may be controlled so as to stop the vehicle at a road position nearest to the passenger.

[0198] According to this embodiment, the destination can be confirmed by other passengers using the information terminal. In addition, by diverting the technology to speed control, it is expected that the accuracy of stopping an autonomous taxi vehicle in front of a passenger can be improved.

[0199] Fourth Embodiment The fourth embodiment is a modified example in which the photographing point on the HMD 1B side, which is the other person's information terminal, is different from the meeting place (FIG. 26).

[0200] In cases where the meeting place is a narrow road off a main road and video captured from the meeting place is not suitable for meeting support, for example, video captured from a position other than the meeting place may be sent to the vehicle-mounted terminal 1A as a comparison video. In this case, since the HMD 1B can grasp its own position in the world coordinate system using the position and orientation sensor 135B, it transmits the coordinate values ​​of the meeting place (target point) in addition to the coordinate values ​​of the shooting point to the vehicle-mounted terminal 1A. The vehicle-mounted terminal 1A can grasp the meeting place by converting the coordinate values ​​of the meeting place in the terminal world coordinate system of HMD 1B into coordinate values ​​in the terminal world coordinate system of the vehicle-mounted terminal 1A using the transformation parameters between coordinate systems obtained by the method of embodiment 1.

[0201] Furthermore, the user of the HMD 1B may be waiting for a taxi inside a building. In this case, the HMD 1B may transmit the coordinate values ​​of a point in front of the building as the meeting place to the vehicle-mounted terminal 1A, and may transmit an image of the building where the user of the HMD 1B is waiting to the vehicle-mounted terminal 1A for confirmation of the meeting place.

[0202] According to this embodiment, video is captured from a shooting position that is different from the meeting place, data necessary for creating external world feature point information is measured, and then the vehicle can move to the original meeting place. Therefore, even if the meeting place is difficult to find because it faces a narrow alley, by setting up a shooting position on a main street, it becomes easier to support a taxi in approaching the meeting place. It is also possible to wait for a taxi at a location away from the meeting place. Note that the external world feature point information in this case is information that directly indicates the relative positional relationship between the external world feature points and the video shooting position, but it can also be said to be information that indirectly indicates the relative positional relationship between the external world feature points and the information terminal.

[0203] Fifth Embodiment In a fifth embodiment, images of external feature points captured by passengers may be provided to a taxi company to be included in a video library. In this case, passengers may be given an incentive, such as a discounted fare. Because the state of the external world may change over time, it is desirable to keep updating the video library with as much new video as possible. This allows the vehicle-mounted terminal 1A to perform location estimation based on the latest situation when using library video, thereby improving the accuracy of location estimation.

[0204] Although the embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain each embodiment, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of each embodiment will not be impaired even if different ones are used.

[0205] The programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each process is performed may also be changed.

[0206] Some or all of the functions of the above-described embodiments may be implemented in hardware, for example, by designing them as integrated circuits. Alternatively, they may be implemented in software by a microprocessor unit, CPU, or the like interpreting and executing an operating program that implements each function. Furthermore, the scope of software implementation is not limited, and hardware and software may be used together. Alternatively, some or all of the functions may be implemented in a server. The server may be any type of server, such as a local server, cloud server, edge server, or internet service, as long as it can execute functions in cooperation with other components via communications. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communications network.

[0207] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all of the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

[0208] The above embodiment includes the following aspects.

[0209] (Supplementary Note 1) An information terminal comprising: a processor; a camera; and a communication interface, wherein the camera captures a first external world image of the vicinity of a meeting place while the information terminal is approaching the vicinity of the meeting place; the processor creates first external world feature point information indicating a relative positional relationship between a first external world feature point in the first external world image and the information terminal based on the first external world image; receives via the communication interface a second external world image captured by an other person's information terminal of the vicinity of the meeting place and second external world feature point information indicating a relative positional relationship between a second external world feature point in the second external world image and the other person's information terminal; compares the first external world image with the second external world image and the first external world feature point information with the second external world feature point information to calculate an inter-terminal positional relationship between the information terminal and the other person's information terminal; and outputs meeting support information for guiding a user of the information terminal to the meeting place based on the calculated inter-terminal positional relationship.

[0210] (Supplementary Note 2) A meeting support method executed in an information terminal, comprising the steps of: generating, by the information terminal, first external world feature point information indicating a relative positional relationship between the information terminal and a first external world feature point in a first external world image, based on a first external world image of the vicinity of the meeting place captured by a camera mounted on the information terminal while approaching the vicinity of the meeting place; receiving, from an other person's information terminal communicatively connected to the information terminal, a second external world image captured of the vicinity of the meeting place and second external world feature point information indicating a relative positional relationship between the information terminal and a second external world feature point in the second external world image; comparing the first external world image with the second external world image, and the first external world feature point information with the second external world feature point information, to calculate an inter-terminal positional relationship between the information terminal and the other person's information terminal; and outputting meeting support information for guiding a user of the information terminal to the meeting place based on the calculated inter-terminal positional relationship.

[0211] (Supplementary Note 3) A meeting support system configured by communicatively connecting an information terminal and another person's information terminal, wherein the information terminal comprises a first camera, a first processor, and a first communication interface; the other person's information terminal comprises a second camera, a second processor, and a second communication interface; the other person's information terminal creates second external world feature point information indicating a relative positional relationship between second external world feature points in the second external world image and the other person's information terminal, based on a second external world image of a location near the meeting place captured by the second camera; and executes control to transmit the second external world image and the second external world feature point information from the second communication interface to the information terminal; the information terminal receives the second external world image and the second external world feature point information via the first communication interface; and the first processor a first external world feature point information indicating a relative positional relationship between the information terminal and a first external world feature point in the first external world image based on a first external world image near the meeting place captured by the first camera while the information terminal is approaching the meeting place; a comparison between the first external world image and the second external world image, and the first external world feature point information and the second external world feature point information, to calculate an inter-terminal positional relationship between the information terminal and the other person's information terminal; and an output of meeting support information for guiding the user of the information terminal to the meeting place based on the calculated inter-terminal positional relationship.

[0212] 1: Vehicle 1A: In-vehicle terminal 1B: HMD 1C: Smartphone 2: Server 3: Network 4: Access point 100: Meeting support system 111A: Outer camera 111B: Outer camera 111C: Outer camera 112C: Inner camera 113A: Distance measurement sensor 113B: Distance measurement sensor 113C: Distance measurement sensor 115A: Acceleration sensor 115B: Acceleration sensor 115C: Acceleration sensor 116A: Gyro sensor 116B: Gyro sensor 116C: Gyro sensor 117A: Geomagnetic sensor 117B: Geomagnetic sensor 117C: Geomagnetic sensor 118A: Positioning sensor 118B: Positioning sensor 118C: Positioning sensor 119A: Display 119B: Display 119C: Display 120A: Network communication device 120B: Network communication device 120C: Network communication device 121A: Microphone 121B: Microphone 121C: Microphone 122A: Speaker 122B: Speaker 122C: Speaker 123A: Antenna 123B: Antenna 123C: Antenna 124: Touch panel 125A: Processor 125B: Processor 125C: Processor 126A: Program 126B: Program 126C: Program 127A: Data 127B: Data 127C: Data 128A: Memory 128B: Memory 128C: Memory 131C: Telephone network communication device 135A : Position and orientation sensor 135B : Position and orientation sensor 135C : Position and orientation sensor 136A : PP camera 136B : PP camera 136C : PP camera 137A : PP distance measurement sensor 137B : PP distance measurement sensor 137C : PP distance measurement sensor 140A : Bus 140B : Bus 140C : Bus 220 : Network communication device 223 : Antenna 225 : Processor 226 : Program 227 : Data 228 : Memory 240 : Bus1000: First external world image 1001: Second external world image 1010A: First external world feature point 1010B: First external world feature point 1010C: First external world feature point 1011A: Second external world feature point 1011B: Second external world feature point 1011C: Second external world feature point 1200: External world image 1210: Virtual object 1211: Virtual object 1300: External world image 1310: Virtual object 1311: Virtual object

Claims

1. An information terminal comprising: a processor; a camera; and a communications interface, wherein the camera captures a first external world image of the vicinity of a meeting place while the information terminal is approaching the vicinity of the meeting place, the processor creates first external world feature point information indicating a relative positional relationship between a first external world feature point in the first external world image and the information terminal based on the first external world image, receives via the communications interface a second external world image captured by an other person's information terminal of the vicinity of the meeting place and second external world feature point information indicating a relative positional relationship between a second external world feature point in the second external world image and the other person's information terminal, compares the first external world image with the second external world image, and the first external world feature point information with the second external world feature point information, and calculates an inter-terminal positional relationship between the information terminal and the other person's information terminal, and outputs meeting support information for guiding a user of the information terminal to the meeting place based on the calculated inter-terminal positional relationship.

2. An information terminal according to claim 1, further comprising a distance measurement sensor and a position and orientation sensor, wherein the camera measures a direction vector of each of a plurality of first external field feature points included in the first external field image as seen from the terminal position of the information terminal, based on a terminal local coordinate system of the information terminal, the distance measurement sensor measures a distance of each of the plurality of first external field feature points from the terminal position of the information terminal, and the position and orientation sensor measures a position and orientation of the information terminal in a first terminal world coordinate system which is a terminal world coordinate system of the information terminal, and the processor calculates coordinate values ​​in the terminal local coordinate system from data on the direction vector and distance as seen from the information terminal for each of the plurality of first external field feature points, and converts the coordinate values ​​of the plurality of first external field feature points in the terminal local coordinate system into coordinate values ​​in the first terminal world coordinate system from the position and orientation data in the first terminal world coordinate system of the information terminal, creating, as the first external environment feature point information, information indicating a relative positional relationship between the information terminal and each of the plurality of first external environment feature points indicated by coordinate values ​​in the first terminal world coordinate system; receiving, as the second external environment feature point information, information indicating a relative positional relationship between the other party's information terminal and each of the plurality of second external environment feature points indicated by coordinate values ​​in a second terminal world coordinate system which is a terminal world coordinate system of the other party's information terminal; searching for all of the plurality of second external environment feature points captured in the second external environment video received from the other party's information terminal to obtain a correspondence relationship between each of the plurality of first external environment feature points and the corresponding second external environment feature point; comparing the first external environment feature point information and the second external environment feature point information corresponding to a pair of the first external environment feature point and the second external environment feature point with which the correspondence relationship is established, and calculating conversion information between the coordinate value in the first terminal world coordinate system and the coordinate value in the second terminal world coordinate system; an information terminal that converts the terminal position of the other party's information terminal expressed in the second terminal world coordinate system into a terminal position expressed in the first terminal world coordinate system using the conversion information to calculate an inter-terminal positional relationship between the information terminal and the other party's information terminal, and generates and outputs the meeting support information based on the calculated inter-terminal positional relationship.

3. An information terminal as described in claim 2, wherein the processor calculates, as the transformation information, a rotation parameter that aligns the orientations of the first terminal world coordinate system and the second terminal world coordinate system from directional vector values ​​in the first terminal world coordinate system and the second terminal world coordinate system of two directional vectors defined in the external world, and calculates a difference between the origins of the first terminal world coordinate system and the second terminal world coordinate system from coordinate values ​​in the first terminal world coordinate system and the second terminal world coordinate system of a position defined in the external world.

4. An information terminal as described in claim 2, wherein the processor generates and outputs the meeting support information based on an average position of estimated positions of multiple other information terminals based on multiple pairs of the first external feature points and the second external feature points that are in the corresponding relationship.

5. An information terminal as described in claim 1, further comprising a display, wherein the processor displays, as the meeting support information, on the display a virtual object indicating the terminal position of the other person's information terminal, superimposed on an external world image including the estimated meeting place, the external world image including the estimated meeting place.

6. An information terminal as described in claim 1, wherein the first external environment feature points are feature points located on a still object included in the first external environment image, and at least three or more of the first external environment feature points are set on the first external environment image, and the second external environment feature points are feature points located on a still object included in the second external environment image, and at least three or more of the second external environment feature points are set on the second external environment image.

7. An information terminal as described in claim 6, wherein, when the terminal position of the other person's information terminal cannot be calculated from the at least three or more first external world feature points, the processor sets new feature points on the first external world image as alternative external world feature points and calculates the terminal position of the other person's information terminal using the alternative external world feature points.

8. An information terminal according to claim 1, wherein the first external feature points are feature points located on a vehicle on which the information terminal is mounted, and at least three of the first external feature points are set on the first external image.

9. A meeting support method executed in an information terminal, comprising the steps of: generating first external world feature point information indicating a relative positional relationship between the information terminal and a first external world feature point in a first external world image based on a first external world image of the vicinity of the meeting place captured by a camera mounted on the information terminal while approaching the vicinity of the meeting place; receiving from an other person's information terminal communicatively connected to the information terminal a second external world image captured of the vicinity of the meeting place and second external world feature point information indicating a relative positional relationship between the information terminal and a second external world feature point in the second external world image and the other person's information terminal; comparing the first external world image with the second external world image, and the first external world feature point information with the second external world feature point information, to calculate an inter-terminal positional relationship between the information terminal and the other person's information terminal; and outputting meeting support information for guiding a user of the information terminal to the meeting place based on the calculated inter-terminal positional relationship.

10. A meeting support system configured by connecting an information terminal and an other person's information terminal via communication, the information terminal comprising a first camera, a first processor, and a first communication interface, the other person's information terminal comprising a second camera, a second processor, and a second communication interface, the other person's information terminal: based on a second external world image of the vicinity of the meeting place captured by the second camera, creates second external world feature point information indicating a relative positional relationship between the other person's information terminal and a second external world feature point in the second external world image; executes control to transmit the second external world image and the second external world feature point information from the second communication interface to the information terminal, the information terminal receives the second external world image and the second external world feature point information via the first communication interface, the first processor: based on a first external world image of the vicinity of the meeting place captured by the first camera while the information terminal is approaching the meeting place, creates first external world feature point information indicating a relative positional relationship between the information terminal and a first external world feature point in the first external world image; a meeting support system that compares the first external world image with the second external world image, and the first external world feature point information with the second external world feature point information, to calculate an inter-terminal positional relationship between the information terminal and the other person's information terminal, and outputs meeting support information for guiding the user of the information terminal to a meeting place based on the calculated inter-terminal positional relationship.

Images