Mobile terminal, terminal position estimation system, and terminal position estimation method

Abstract

To accurately estimate, with a simple configuration, the current position and orientation of a user even in a location where radio waves from a satellite, etc., do not reach.SOLUTION: The current position and orientation of a user can be accurately estimated with a simple configuration even in a location where radio waves from a satellite, etc., do not reach. A mobile terminal 100 comprises a camera 121 and a processing unit for processing an image acquired by the camera 121. The processing unit comprises: an image acquisition unit 171 which uses the camera 121 to acquire a sign image that is an image including different two sign points that are located at positions separated from the mobile terminal 100 and have known position information; and a calculation unit 172 which analyzes the acquired sign image, calculates a planar orientation and interior angles of a first triangle formed by the two sign points and a terminal position that is the position of the mobile terminal 100, and uses the planar orientation and interior angles to calculate the terminal position.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a position and direction information estimation technology, and in particular to a technology for estimating position and direction information that is not received by radio waves from artificial satellites, etc. This paper relates to a technology for estimating position and direction information in difficult locations. [Background technology]

[0002] Even in places where radio waves from satellites cannot reach, current location information and direction information can be obtained ( For example, Patent Document 1 states that "Latitude and longitude information is set for multiple buildings. A plurality of location information display devices for placing a vehicle, and latitude and longitude information written on the location information display devices The position information reading device reads the position information from the position information display device that is closest to the current position. The current location information is obtained by reading the latitude and longitude information using a location information reader. The system is equipped with a direction information display device that is installed near the latitude and longitude information, and displays the direction information most suitable for the current position. Acquires azimuth information by reading azimuth information from a nearby azimuth information display device (abstract excerpt) )" technology has been disclosed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2000-205888 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 obtains location information from a pre-installed location information display board. The location information is not available for places without location information boards. For example, in a large underground mall or shopping center, the current location and direction cannot be obtained. When trying to find a destination such as a store by determining the location, errors are likely to occur.

[0005] The present invention has been made in consideration of the above circumstances, and is intended to provide a method for controlling a wireless communication system in a place where radio waves from an artificial satellite or the like cannot reach. To provide a technology for estimating a user's current position and direction with high accuracy using a simple configuration. The purpose is to [Means for solving the problem]

[0006] The present invention relates to Mobile A mobile terminal, The device is equipped with a camera, a device position estimation unit that processes an image acquired by the camera, a three-axis acceleration sensor, a three-axis gyro sensor, a clock unit, and a distance measurement sensor, and the camera captures an image of a signboard that is attached or installed on the surface of a landmark object whose position information is known to generate a sign image, and the signboard includes position information and orientation information of the signboard, and a landmark point used to measure the position of the mobile device, and the device position estimation unit includes an image acquisition unit that acquires the sign image, analyzes the acquired sign image, and generates a terminal position estimation unit that estimates the side of the side having a length corresponding to the distance from the mobile device to the signboard measured by the distance measurement sensor and the position of the mobile device. and the landmark, and a calculation unit that calculates a plane orientation of a triangle including the landmark and the landmark and an interior angle of the triangle, and calculates the terminal position relative to the signboard using the plane orientation and the interior angle, wherein the calculation unit detects the direction and amount of movement from the terminal position when the signboard was imaged based on the distance and direction of movement detected by the three-axis acceleration sensor, the three-axis gyro sensor, and the clock unit, to estimate the terminal position, and if it is determined that an error in the position estimation has become large, the calculation unit indicates the position of the signboard near the mobile terminal to prompt the user of the mobile terminal to measure the position, or the mobile terminal automatically measures the position. The present invention also provides a mobile terminal comprising a camera, a terminal position estimation unit that processes an image acquired by the camera, a three-axis acceleration sensor, a three-axis gyro sensor, and a clock unit, the camera executes a function of calculating a focal length, the camera captures an image of a signboard attached or installed on a surface of a landmark object whose position information is known, and generates a sign image, the signboard including position information of the signboard, orientation information of the signboard, and a landmark point used to measure the position of the mobile terminal, the terminal position estimation unit includes an image acquisition unit that acquires the sign image, , and a calculation unit that analyzes the acquired sign image, calculates a surface orientation of a triangle including a side having a length corresponding to the distance from the mobile device to the sign board calculated based on a focal length when the camera captured the sign board, a terminal position which is the position of the mobile device, and the sign point, and an interior angle of the triangle, and calculates the terminal position relative to the sign board using the surface orientation and the interior angle, wherein the calculation unit detects a moving direction and a moving amount from the terminal position when the sign board was captured based on a moving distance and a moving direction detected by the three-axis acceleration sensor, the three-axis gyro sensor, and the clock unit, and estimates a terminal position, and when it is determined that an error in the position estimation has increased, indicates a position of the sign board near the mobile device to prompt a user of the mobile device to measure the position, or causes the mobile device to automatically measure the position. It is characterized by:

[0007] The present invention also provides a method for producing a method for manufacturing a semiconductor device comprising the steps of: A terminal position estimation system comprising a mobile terminal equipped with a camera and a server that processes images acquired by the camera, and that estimates a terminal position, which is the position of the mobile terminal, by analyzing the images, wherein the mobile terminal is equipped with a 3-axis acceleration sensor, a 3-axis gyro sensor, a clock unit, and a focal length calculation function unit, the camera captures an image of a signboard attached or installed on the surface of a landmark object whose position information is known to generate a sign image, the signboard includes position information and orientation information of the signboard, and a landmark point used to measure the terminal position, and the server comprises an image acquisition unit that acquires the sign image acquired by the camera, and a server that processes images acquired by the camera and processes the images acquired by the camera according to the focal length at which the camera captured the signboard. and a calculation unit that calculates a plane orientation of a triangle including the terminal position and the landmark, a side having a length corresponding to the distance from the mobile terminal to the signboard calculated based on the calculated distance, and an interior angle of the triangle, and calculates the terminal position relative to the signboard using the plane orientation and the interior angle, wherein the calculation unit detects a moving direction and an amount of moving from the terminal position when the signboard was imaged based on the moving distance and moving direction detected by the three-axis acceleration sensor, the three-axis gyro sensor, and the clock unit, and estimates a current terminal position, and when it is determined that an error in the position estimation has increased, the calculation unit indicates the position of the signboard near the mobile terminal to prompt the user of the mobile terminal to measure the position, or the server automatically performs the position measurement. Characterized by

[0008] Furthermore, the present invention provides a method for producing a A terminal that estimates the terminal location, which is the location of a mobile terminal, by using a server that processes images captured by a camera mounted on the mobile terminal. Location estimation The method includes the steps of: acquiring a sign image generated by capturing an image of a signboard, the signboard being attached or installed on a surface of a landmark object whose position information is known, the signboard including position information and orientation information of the signboard and a landmark point used for measuring the position of a mobile device, with a camera mounted on the mobile device; analyzing the acquired sign image, and calculating a terminal position, which is the position of the mobile device, and a terminal position, which is the position of the mobile device, corresponding to a side having a length corresponding to a distance from the mobile device to the signboard measured by a distance measuring sensor mounted on the mobile device, or a distance from the mobile device to the signboard calculated based on a focal length when the camera captured the image of the signboard; the step of calculating the plane orientation of a triangle including the target point and an interior angle of the triangle, and calculating the terminal position relative to the signboard using the plane orientation and the interior angle, the step of detecting the direction and amount of movement from the terminal position when the signboard was imaged based on the distance and direction of movement detected by a three-axis acceleration sensor, a three-axis gyro sensor, and a clock unit mounted on the mobile terminal, and estimating the current terminal position, and if it is determined that an error in the position estimation has increased, the step of indicating the position of the signboard near the mobile terminal and prompting the user of the mobile terminal to measure the position. Mobile 2. A method for estimating a terminal position, which is the position of a mobile terminal, by using a server that processes an image acquired by a camera mounted on the mobile terminal, the method comprising the steps of: acquiring a sign image generated by an image capturing device mounted on the mobile terminal of a signboard attached or installed on a surface of an object whose position information is known, the signboard including position information and orientation information of the signboard and a landmark point used for measuring the position of the mobile terminal; analyzing the acquired sign image and estimating a distance from the mobile terminal to the signboard measured by a distance measuring sensor mounted on the mobile terminal, or a distance from the mobile terminal calculated based on a focal length when the camera captured the image of the signboard. the step of calculating a plane orientation of a triangle including the mobile terminal position and the landmark point, a side of which has a length corresponding to the distance from the landmark to the signboard, and an interior angle of the triangle, and calculating the terminal position relative to the signboard using the plane orientation and the interior angle; the step of detecting the direction of movement and the amount of movement from the terminal position when the signboard was imaged based on the distance and direction of movement detected by a three-axis acceleration sensor, a three-axis gyro sensor, and a clock unit mounted on the mobile terminal, to estimate the current terminal position; and the step of the server automatically measuring the position if it is determined that an error in the position estimation has become large. Effect of the Invention

[0009] Even in places where radio waves from satellites cannot reach, the system can accurately and accurately measure the user's current situation with a simple configuration. The present position and direction can be estimated. In addition to the above, the following problems, configurations and effects can be achieved. This will become clear from the description of the embodiments. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is an explanatory diagram for explaining an overview of a process according to the first embodiment; [Diagram 2] FIG. 2 is an explanatory diagram for explaining an example of a signboard according to the first embodiment. [Diagram 3] FIG. 2A is a hardware configuration diagram of a mobile terminal according to the first embodiment, and FIG. 2B is a functional block diagram of a terminal position estimation unit of the mobile terminal according to the first embodiment. [Figure 4] FIG. 1A is an explanatory diagram for explaining a state during terminal position estimation in the first embodiment, and FIG. 1B is an explanatory diagram for explaining a general position calculation method using triangulation. [Diagram 5] 5(a) to 5(d) are diagrams illustrating a method for calculating an angle between a mobile terminal and a sign surface according to the first embodiment. [Figure 6] FIG. 2 is an explanatory diagram for explaining a terminal position estimation method according to the first embodiment; [Figure 7] 4 is a flowchart of a terminal position estimation process according to the first embodiment. [Figure 8] 13(a) and (b) are explanatory diagrams for explaining a modified example of the signboard of the first embodiment. FIG. [Figure 9] 13(a) and 13(b) are explanatory diagrams for explaining a terminal position estimation method according to a modified example of the first embodiment. [Figure 10] FIG. 11 is an explanatory diagram for explaining a terminal position estimation method according to a modified example of the first embodiment. [Figure 11] 13A and 13B are diagrams illustrating an example of a service and a display screen provided according to a modification of the first embodiment. [Figure 12] FIG. 11 is an explanatory diagram for explaining an example of a signboard according to a second embodiment. [Figure 13] 13(a) to 13(c) are explanatory diagrams for explaining a terminal position estimation method according to a second embodiment. [Figure 14] 13A and 13B are explanatory diagrams illustrating a terminal position estimation method according to a second embodiment. [Figure 15] 13(a) to 13(c) are explanatory diagrams for explaining a terminal position estimation method according to a second embodiment. [Figure 16] 13A and 13B are explanatory diagrams illustrating a terminal position estimation method according to a second embodiment. [Figure 17] 13(a) and (b) are explanatory diagrams showing the three-dimensional position of a target point. [Figure 18]FIG. 13 is an explanatory diagram for explaining an example of a landmark according to the third embodiment. [Figure 19] 13(a) to 13(c) are explanatory diagrams illustrating a terminal position estimation method according to a third embodiment. [Figure 20] 13(a) and 13(b) are explanatory diagrams for explaining a terminal position estimation method according to a third embodiment. [Figure 21] 13(a) to 13(c) are explanatory diagrams illustrating a terminal position estimation method according to a modified example of the third embodiment. [Figure 22] 13(a) to 13(d) are explanatory diagrams for explaining an example of the positional relationship between landmarks and additional landmarks in a modified example of the third embodiment. [Diagram 23] 13(a) and (b) are explanatory diagrams for explaining a terminal position estimation method according to a modified example of the third embodiment. [Figure 24] 13(a) and (b) are explanatory diagrams for explaining a terminal position estimation method according to a modified example of the third embodiment. [Diagram 25] FIG. 13 is an explanatory diagram for explaining a terminal position estimation method according to a modified example of the third embodiment. [Figure 26] 13(a) to 13(c) are explanatory diagrams illustrating a terminal position estimation method according to a modified example of the third embodiment. [Figure 27] FIG. 13 is an explanatory diagram for explaining a terminal position estimation method according to a modified example of the third embodiment. [Figure 28] 13(a) and (b) are explanatory diagrams for explaining a terminal position estimation method according to a modified example of the third embodiment. [Figure 29] 13(a) and (b) are explanatory diagrams for explaining a terminal position estimation method according to a modified example of the third embodiment. [Diagram 30] 13(a) and 13(b) are diagrams illustrating estimated position data in an application example of the third embodiment. [Diagram 31] 13 is a flowchart of a terminal position estimation process in an application example of the third embodiment. [Diagram 32] FIG. 13 is an explanatory diagram for explaining an estimated position history in an application example of the third embodiment. [Diagram 33] 13 is a flowchart of a terminal position estimation process in an application example of the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the above, unless otherwise specified, the same reference numerals are used to denote components having the same functions, and repeated explanations will be omitted. Omitted.

[0012] <<First embodiment>> In this embodiment, the location information is a known landmark, and the location is a distance from the camera. The landmark is photographed with a camera and the camera's position is calculated (estimated) through image processing.

[0013] In the following description of the present embodiment, landmarks with known position information are set in advance as objects such as pillars and walls. The signboard is a signboard attached or installed on a surface. The signboard shall include the location information of the signboard and the sign It is assumed that the orientation information of the signboard is registered. In this embodiment, the signboard is The shape of the signboard must enable the angle between the line segment connecting the center of the signboard and the surface of the signboard to be grasped.

[0014] In this embodiment, two sign boards are photographed, and the position information of each sign board and the surface of each sign board and the camera are recorded. The camera position is calculated based on the angle between the position and the center of the signboard and the direction information of each signboard. The position information of the device is calculated.

[0015] FIG. 1 is a diagram for explaining an outline of the processing of this embodiment. As shown in this figure, A person 910 is a portable information processing device (hereinafter, referred to as a portable terminal) equipped with a photographing function (camera). ) 100 and takes a picture of the sign board 200.

[0016] The mobile terminal 100 is connected to a preset access point (AP) 970 and a network A server 960 is connected to the network 940 or a base station 950 of a mobile phone company. It is possible to connect to.

[0017] [Sign board] The details of the sign board 200 will be described with reference to FIG. The plate 200 has, for example, a square shape with a side length of LS.

[0018] As shown in FIG. 2, the signboard 200 stores position information and direction information (hereinafter, both of them are referred to as “information”). This is called position and direction information. The position and direction information display area 201 is displayed in a readable manner by the Q The R code (registered trademark) may include position and direction information. In the example shown in the figure, the sign 2 Position and orientation information is written on the 00 sign surface.

[0019] The position information is, for example, the latitude and longitude of the center point of the sign board 200. The direction information is, for example, The horizontal arrow 202 on the sign face of the sign board 200 (hereinafter referred to as the directional arrow 202) , and is indicated by an angle with a predetermined reference direction. The direction and the angle formed shall be registered.

[0020] The contents of the position and direction information display area 201 can be visually confirmed by the owner 910. The sign board 200 may be photographed by a camera at the terminal 100 and the image may be acquired through image processing.

[0021] The position and direction information does not have to be clearly indicated on the signboard 200. For example, The URL of the information source is written on the smartphone or tablet (registered trademark) and by taking a picture with the camera, The position and orientation information of the sign board 200 may be acquired from a server or the like.

[0022] [Mobile device configuration] Next, a configuration of the mobile terminal 100 will be described. The mobile terminal 100 has a photographing function and an information processing function. For example, a mobile phone, a smartphone, or a tablet terminal. , wearable devices such as watches and head-mounted displays, and feature phones , or other handheld digital devices. In addition, autonomous devices such as drones good.

[0023] FIG. 3(a) is a hardware configuration diagram of the mobile terminal 100 of this embodiment. As shown in the figure, the mobile terminal 100 includes a CPU (Central Processing Unit). t) 101, a storage device 110, a photographing device 120, and a user interface (I / F) 130, a sensor device 140, a communication device 150, and an expansion I / F 160. The device is connected to a CPU 101 via a bus 102 .

[0024] The CPU 101 is a microprocessor unit that controls the entire mobile terminal 100 . The bus 102 is used for transmitting and receiving data between the CPU 101 and each device of the mobile terminal 100. It is a data communication channel.

[0025] The storage device 110 includes a ROM (Read Only Memory) 111 and a RAM ( The system includes a random access memory (RAM) 112 and a storage 113.

[0026] ROM111 stores basic operating programs such as the operating system and other operating This is a memory in which programs are stored. The ROM 111 may be, for example, an EEPROM (EEPROM). Lectrically Erasable and Programmable Re Rewritable ROM such as Ad Only Memory or Flash ROM is used. I can.

[0027] The storage 113 stores the operation program and operation setting values ​​of the mobile terminal 100, It stores various programs and data necessary to realize functions.

[0028] The storage 113 stores the information even when the mobile terminal 100 is not receiving power from an external source. The storage 113 holds information that is stored in the storage device 112. For example, a flash ROM or an SSD (S Solid State Drives, HDDs (Hard Disk Drives), etc. A device is used.

[0029] RAM 112 is a work area for executing basic operation programs and other operation programs. It is.

[0030] The ROM 111 and the RAM 112 may be integrated with the CPU 101. In addition, the ROM 111 is not an independent structure as shown in FIG. A part of the storage area may be used. This may substitute for all or part of the functions of ROM 111.

[0031] Each operation program stored in the ROM 111 or the storage 113 is, for example, Updates and feature expansions can be performed by downloading from each distribution server on the network. can.

[0032] The photographing device 120 includes a camera 121, an image processor 122, an image memory 123, and Equipped with.

[0033] The camera 121 is a CCD (Charge-Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor ) sensor to convert the light input from the lens into an electrical signal. The surroundings and objects are captured as image data.

[0034] The image processor 122 performs frame filtering on the image data acquired by the camera 121 as necessary. Format conversion, menus and other OSD (On-Screen Display) signals The image processor 122 is provided with a video RAM (not shown) and performs image processing. The video RAM drives a display 131, which will be described later, based on the image data input to the video RAM. The image processor 122 is, for example, a codec that compresses and expands the captured images and video. a picture quality improvement processing section that improves the picture quality of images and videos, and a picture quality correction processing section that corrects the angle and rotation of the captured image. It also realizes functions such as an image processing unit that recognizes information such as QR codes and performs corrections. .

[0035] The image memory 123 stores image data acquired by the camera 121 or the image processor 122. The processed image data is temporarily stored in the memory.

[0036] The user I / F 130 includes, for example, a display 131. The display 131 includes For example, the display unit is a display device such as a liquid crystal panel, and each part of the mobile terminal 100 serves as a display unit. The operation unit also functions as a reception unit that receives input of operation instructions. The display device may have a touch panel function in which the display devices are arranged on top of each other.

[0037] The functions of the controller can be controlled by a keyboard, mouse, or other device connected via the expansion I / F 160 (described later). This may also be realized by a mouse, etc.

[0038] The sensor device 140 is a group of sensors for detecting the state of the mobile terminal 100. For example, GPS (Global Positioning System) receiver The device includes a receiver 141, a three-axis gyro sensor 142, and a three-axis acceleration sensor 143. In addition, distance sensors, geomagnetic sensors, illuminance sensors, proximity sensors, biometric sensors, and atmospheric pressure sensors The device may also include a spring.

[0039] These sensors detect the position, inclination, direction, movement, etc. of the mobile terminal 100. If the current location is within a location where GPS signals can be received, the location information will be sent via GPS reception. The signal is acquired by the device 141.

[0040] The communication device 150 performs communication between the mobile terminal 100 and an external device. For example, a LAN (Local Area Network) communication device 151 and a telephone network communication device The device 152 and the short-range communication device 153 are provided.

[0041] The LAN communication device 151 is an access point via a wireless connection such as Wi-Fi (registered trademark). It is connected to the network via an AP (Access Point) device and shares data with other devices on the network. Send and receive.

[0042] The telephone network communicator 152 communicates with a base station of a mobile telephone network by radio communication, and transmits and receives voice and data. Send and receive.

[0043] The short-range communication device 153 is, for example, a USB (Universal Serial Bus The mobile terminal 100 transmits and receives data to and from other devices in the vicinity of the mobile terminal 100 via a wired connection means such as a wireless LAN cable. In addition, data is transmitted and received via wireless communication with other devices equipped with short-range communication devices. The transmitter 153 is, for example, a near field communication (NFC) It is an interface for the ICT, and is extremely short, ranging from a few centimeters to approximately 10 centimeters. Two-way communication between devices equipped with NFC chips may be achieved at a distance. Supports services using contactless IC chips such as electronic money that will be installed in smartphones at the end of the year In addition, the short-range communication device 153 transmits data to other devices equipped with a wireless communication device by wireless communication. For example, Bluetooth (registered trademark) can be used to transmit and receive signals at distances of several meters to several meters. It uses radio waves to enable simple information exchange between information devices at a distance of about 10 meters.

[0044] The LAN communication device 151, the telephone network communication device 152, and the short-range communication device 153 are respectively: The communication device 150 includes an encoding circuit, a decoding circuit, an antenna, etc., and realizes infrared communication. The communication device may further include a communication device for communicating with the communication device.

[0045] The expansion I / F 160 is a group of interfaces for expanding the functions of the mobile terminal 100. In this embodiment, the device is equipped with a video / audio I / F, an operating device I / F, a memory I / F, etc. The / audio I / F is for inputting video / audio signals from external video / audio output devices, Outputs video and audio signals to audio input devices, etc. External operation devices such as keyboards The memory interface is connected to the memory card and other memory. It connects the medium to send and receive data.

[0046] The configuration example of the mobile terminal 100 shown in FIG. 3(a) focuses on the essential configuration of this embodiment. The mobile terminal 100 has the above configuration, and also has a digital broadcast receiving function and an electronic It may also include additional components not shown, such as a money settlement function.

[0047] Next, the functions of the mobile terminal 100 will be described. The main focus is on the configuration related to the terminal location estimation function (terminal location estimation unit) that calculates the current location information. As shown in FIG. 3(b), the mobile terminal 100 of this embodiment performs terminal position estimation. The determination unit 170 includes an image acquisition unit 171, a calculation unit 172, a display control unit 173, and a data A storage unit 174 is provided.

[0048] The image acquisition unit 171 controls the photographing device 120 to acquire an image. Furthermore, the acquired image is subjected to image compression / expansion processing, image improvement processing, etc. In this case, the image of the sign board 200 is acquired. More specifically, the image of the sign board 200 at two different locations is acquired. Each will be obtained.

[0049] The calculation unit 172 analyzes the image acquired by the image acquisition unit 171 and calculates the current position of the mobile terminal 100. Calculate the position.

[0050] The calculation unit 172 calculates the contents of the sign board 200 using various correction functions and character recognition techniques, for example. The direction and angle of the sign board 200 are identified based on the shape of the sign board 200 that has been analyzed or photographed. Or digitize it.

[0051] The display control unit 173 controls the display on the display 131 .

[0052] The data storage unit 174 stores data necessary for processing, data in progress and results of processing, and data to be generated. Stores the data.

[0053] In addition, when the position and direction information is registered in the signboard 200 as a QR code, the QR The photographing device 1 may further include a QR code analyzing unit 175. The content of the QR code photographed in 20 is analyzed. The analysis result is displayed on the display control screen. The information may be displayed on the display 131 by the unit 173.

[0054] Each unit of the terminal position estimation unit 170, except for the data storage unit 174, is controlled by the CPU 101. A program stored in the storage 113 or the ROM 111 is expanded into the RAM 112. However, the above software is not entirely software. For example, a part of the data may be implemented as hardware to increase the speed. The memory unit 174 is constructed in the storage device 110 .

[0055] [Device location estimation] Next, the terminal position estimation process performed by the terminal position estimation unit 170 of this embodiment will be described. The image acquisition unit 171 analyzes the information of the sign boards 200 at two different locations, and calculates the image. 72 calculates the current location of the owner 910 (terminal location estimation) as shown in FIG. In this embodiment, the current location is calculated based on the location of the owner 910. In the following, in this embodiment, the calculated current location is referred to as the terminal This is called the end position.

[0056] Here, as shown in FIG. 4(a), an owner 910 of a mobile terminal 100 that he or she owns The following description will be given by taking an example in which the camera 121 is used to capture images of sign boards 211 and 212. The mobile terminal 100 analyzes the captured image and calculates the terminal position.

[0057] Hereinafter, the center position of the sign board 211 is point A, the center position of the sign board 212 is point B, and the terminal position is point C. In this embodiment, points A, B, and C are on the same horizontal plane. If the terminals are not on the same horizontal plane, an error will occur in the estimated terminal position. This method can be used within the permissible range. Hereafter, this plane is called the x, y plane. The coordinates of point A are (xa, ya), the coordinates of point B are (xb, yb), and the coordinates of point C are (xc, yc In this embodiment, the sign board 211 and the sign board 212 are photographed to obtain the points. Angle α1 between point C and the sign surface of the sign board 211 and angle α2 between point C and the sign surface of the sign board 212 The coordinates of points A and B are actually measured. is expressed in latitude and longitude, but is converted to coordinates on the x, y plane mentioned above before calculation.

[0058] Here, the terminal position C is calculated by triangulation. The point you want to measure can be determined by measuring the angle from a known point to the point you want to measure. It is a surveying method that uses trigonometry and geometry to calculate the position of a point.

[0059] Here we will briefly explain the method of calculating the position of a point to be surveyed using triangulation. The position calculation method using triangulation is briefly explained with reference to FIG. 4(b). The quantity is formed by point C, the position of which is to be found, and two points (point A and point B), the positions of which are known. In a triangle, what is the angle between line segment CA and line segment AB, and what is the angle between line segment BC and line segment AB? The position of point C is calculated by measuring the degrees α and β.

[0060] In FIG. 4(b), we consider a coordinate system in which the direction connecting points A and B is the x-axis direction. Let L be the length of the line segment AB, d be the distance between point C and the line segment AB, and the angle between line segments CA and AB. Let the end angles between line segment BC and line segment AB be α and β, respectively.

[0061] If the coordinates of point A are A(xa,ya), then the coordinates of point B are B(xa+L,ya). And the coordinates (xc, yc) of point C are expressed by the following equation. xc = xa + d / tan α (1) yc=ya-d (2) Here, d and L are calculated using the following formula. d=L / (1 / tanα+1 / tanβ) ···(3) =L·sinα·sinβ / sin(α+β) ···(4)

[0062] By substituting this d into the coordinates of point C, the coordinates xc and yc of point C become L and It is expressed using α and β.

[0063] In this embodiment, as described above, the image acquisition unit 171 acquires images of the sign boards 2 at two different locations. Images (labeled images) 221 and 222 of 11 and 212, respectively, are acquired.

[0064] The calculation unit 172 analyzes the sign images 221 and 222 acquired by the image acquisition unit 171, The position information and both end angle information of the sign boards 211 and 212 are calculated. , a triangle with the center point of the sign board 211, the center point of the sign board 212, and the terminal position as vertices, The angles at both ends of the side connecting the sign boards 211 and 212, i.e., the angles corresponding to the above-mentioned angles α and β, be.

[0065] In the following, in this embodiment, the sign board 200 and the mobile terminal 100 are placed on a horizontal plane. The intersection of the 00 sign surface and the horizontal plane and the line connecting the mobile terminal 100 and the center point of each sign surface form The angle is simply referred to as the angle between the sign surface and the mobile terminal 100.

[0066] The position information of the sign boards 211 and 212 is obtained from the sign images 221 and 222. The value is obtained by reading

[0067] For example, if the location and direction information is written in letters and numbers, it can be recognized by OCR (Optical Character Recognition). By encoding it using the Character Recognition (Character In addition, if the location and direction information is written in a QR code, etc., the QR code is read and By reading the signal, position and direction information is obtained.

[0068] The angles α1 and β1 are determined based on the size of the sign images 221 and 222 and the direction information of the position and direction information. Specifically, the ratio of the length of the vertical side to the length of the horizontal side of the sign image 221 and the orientation are used. Use information and.

[0069] Here, as shown in FIG. 5(a), the sign board 200 is a square with a side length of LS0. As shown in FIG. 5(b), on a horizontal plane including the signboard 200 and the mobile terminal 100, When the angle between the sign surface of the identification plate 200 and the mobile terminal 100 is α1, as shown in FIG. As shown in FIG. 1, the ratio of the vertical side LS1 to the horizontal side LS2 of the sign image 221 captured by the mobile terminal 100 is s In addition, when the angle α1 between the sign surface and the mobile terminal 100 is 90°, As shown in 5(c), sin(90°) = 1.

[0070] For example, as shown in FIG. 6, when the angle between the mobile terminal 100 and the sign surface of the sign board 211 is α 1. Also, the angle between the mobile terminal 100 and the sign surface of the sign board 212 is assumed to be β1.

[0071] Here, the direction of the arrow indicating the direction on the signboard 211 (the angle that the arrow on the signboard surface makes with north) Then, α2 is registered, and β2 is registered as the direction of the arrow indicating the direction of the sign board 212. It is assumed that there is.

[0072] Furthermore, the angle that the direction from the sign board 211 to the sign board 212 makes with the east-west direction is defined as γ2. In addition, γ2 is calculated from the latitude and longitude information of the sign board 211 and the sign board 212. That is, it is calculated by the following formula: L = √{(xa-xb) 2 +(ya-yb) 2} ···(5) tan(γ2)=(yb-ya) / (xb-xa) ···(6)

[0073] In this case, as shown in FIG. 6, the following relationship exists between α, α1, α2, and γ2: α1+α+(90 degrees - α2)+γ2=180 degrees...(7) Therefore, α = 90° - α1 + α2 - γ2 (8) Furthermore, the following relationship exists between β, β1, β2, and γ2: β1+β+(90 degrees-γ2-β2)=180 degrees...(9) Therefore, β=90 degrees-β1+β2+γ2...(10)

[0074] In this way, the position information (xa, ya) and the direction information α2 of point A and the position information (xb , yb), azimuth information β2, and an angle α1 between the sign surface of the sign board 211 and the mobile terminal 100. If the angle β1 between the sign surface of the sign board 212 and the mobile terminal 100 is known, the distance between point A and point B can be calculated. Distance L, angle α between line segment CA and line segment AB, and angle β between line segment CB and line segment AB. Therefore, the position information (xc, yc) of point C can be obtained.

[0075] Next, the flow of the terminal position estimation process by the image acquisition unit 171 and the calculation unit 172 of this embodiment will be described. This will be described. FIG. 7 shows a process flow of the terminal position estimation process of this embodiment. It is started by an instruction from the owner 910. For example, the mobile terminal 100 has multiple modes. In this case, when an instruction to switch to a mode in which the terminal position estimation process of the present embodiment is executed is received, This may be triggered by the above.

[0076] The owner 910 is in a position where he can take pictures of the two different signs 211 and 212. Then, the process starts.

[0077] The image acquisition unit 171 acquires the sign image of the first sign board 211 in accordance with an instruction from the owner 910. An image 221 is acquired (step S1101).

[0078] The calculation unit 172 analyzes the sign image 221 and obtains the position and orientation information of the first sign board 211. In this case, the position and direction information display area 20 on the sign image 221 is 1, and the position information of the center position of the first sign board 211 and the azimuth arrow are output. Get the orientation of 202.

[0079] The calculation unit 172 also analyzes the sign image 221 and calculates the angle between the sign surface and the mobile terminal 100. α1 is calculated (step S1103).

[0080] Next, the image acquisition unit 171, in accordance with an instruction from the owner 910, The sign image 222 is acquired (step S1104).

[0081] The calculation unit 172 analyzes the sign image 222 and obtains the position and orientation information of the second sign board 212. In this case, the position and direction information display area 20 on the sign image 222 is The area corresponding to 1 is image-processed, and the position information of the center position of the second sign board 212 and the azimuth arrow are obtained. Get the orientation of 202.

[0082] The calculation unit 172 also analyzes the sign image 222 and calculates the angle between the sign surface and the mobile terminal 100. β1 is calculated (step S1106).

[0083] The calculation unit 172 calculates the position and orientation information and angle of the first sign board and the position and orientation of the second sign board. Using the information and the angle, the terminal position, which is the current position of the mobile terminal 100, is calculated using the above method. (step S1107).

[0084] The calculation unit 172 stores the calculation result in the data storage unit 174 and also transmits the result to the display control unit 17 The calculation result is displayed on the display 131 (step S1108), and the process ends. do.

[0085] As described above, in this embodiment, two different positions are located away from the mobile terminal 100. A landmark image is obtained, which is an image including landmarks having known position information. The marker image is analyzed, and a marker formed by the terminal position C, which is the position of the mobile terminal 100, and the two marker points is obtained. A plane orientation of the first triangle and an interior angle of the first triangle are calculated, and the plane orientation and the interior angle are calculated. In this embodiment, the terminal position C is calculated using the marker image. Two different sign boards 200 whose center points are on the same horizontal plane are photographed. The center points of the sign board 200 are set as two sign points. The sign board 200 has two sign points. The position information and the orientation of the sign surface relative to the reference direction can be read by analyzing the sign image. The position and direction information display area 201 is provided on the sign surface, which is the surface of the sign board 200. From the distortion of the sign, the angle that the line segment connecting the camera 121 and the center point makes with the normal line of the sign surface can be measured. In the present embodiment, the calculation unit calculates, from the sign image, the inner angles of the first triangle as follows: Calculate the angles of both ends of the line segment connecting the two landmarks. Then, use triangulation to determine the terminal position C. Calculate.

[0086] In this way, according to the present embodiment, by simply photographing the two sign boards 200, Therefore, according to this embodiment, the vehicle can be easily located in an underground mall or a large shopping mall. This makes it possible to obtain accurate location information even in places where GPS functions cannot be used, such as inside a training center. .

[0087] In addition, the signboard 200 is provided with a so-called QR code that allows access to information such as advertisements in addition to the location information. In this case, the mobile terminal 100 uses the When capturing an image, the QR code can also be read. It can also display available information based on the mode.

[0088] A typical QR code can hold up to 7,089 numbers or 1,817 kanji or kana characters. Therefore, with this configuration, terminal position X At the same time, the names and locations (longitude, latitude, Displaying the location on a detailed map, providing information on products and events handled by the store, etc. Furthermore, the location information of nearby signboards may be included.

[0089] <Modification 1 of the first embodiment> In the above embodiment, the sign board 200 is described as being square. The sign board 200 is square in shape to simplify the calculation of the angle between the terminal position and the sign surface. be.

[0090] However, the sign board 200 may be difficult to see due to distortion in the captured image of the sign board 200. A line connecting the position of the mobile terminal 100 equipped with the camera 121 that took the photograph and the center of the signboard 200 is The angle measuring device has an angle measuring mechanism capable of measuring the angle between the normal line of the sign surface, which is the surface of the sign board 200, and the sign surface. That's good.

[0091] For example, the shape of the sign board 200 is determined by using a photographed sign image to determine whether the camera position matches the sign surface. Any shape can be used as long as the angle between the long side and the short side can be calculated. For example, a rectangle or a circle with a known ratio of the long side to the short side can be used. , an ellipse with a known ratio of its major axis to its minor axis, etc.

[0092] For example, an example in which the sign board 200a is circular is shown in FIG. The diameter of the circle is f. When this signboard 200 is viewed diagonally on the same horizontal plane, the height The length in the vertical direction is f, but the length in the horizontal direction is shorter than f. This aspect ratio is , varies depending on the angle between the signboard 200 and the mobile terminal 100.

[0093] For example, as shown in FIG. 5(b), when the sign board 200 is photographed from the direction of angle α1, The following equation holds: sinα1=f1 / f (11) Note that f1 is the horizontal length when the vertical length is f.

[0094] Therefore, in this case, the calculation unit 172 analyzes the acquired sign image and calculates the angle according to the above formula. Calculate the degree α1.

[0095] In addition, the shape of the signboard 200 itself does not matter. For example, as shown in FIG. 8(b), The four vertices of the rectangle are marked with reference points (the points where the marks shown as crosses in the figure exist) 213. In this case, the four reference points 213 may be determined by dividing the vertical and horizontal lengths of the four reference points 213 by the number of reference points 213. The ratio of the length of the direction is known.

[0096] In addition, the shape of the sign board 200 itself does not have to have the above characteristics. It is sufficient to provide a mark having the above characteristics on the mark surface of 00. For example, a rectangular frame, a circular frame, etc.

[0097] <Modification 2 of the first embodiment> In addition, when calculating the terminal position, the focal length calculation function of the distance measurement sensor and the camera 121 is used. The calculation method in this case will be described with reference to Figs. 9 and 10. When using the focal length calculation function of the camera 121, the range in which the camera 121 can calculate the focal length is The area is limited.

[0098] First, the principle of calculation will be explained. Here, as in Figure 4(b) above, three points, A, B, and C, are used. In a triangle ABC with vertices 1 and 2, what are the positions of points A and B and the length (distance) of side AC, LB? When the length (distance) of side BC and the length (distance) LA of side BC are known, the position of point C can be calculated. In this embodiment, the distance LB and the distance LA are measured by the distance measuring sensor or the camera 121. The focal length is obtained using the focal length calculation function.

[0099] Here, among the interior angles of triangle ABC, the angle between sides CA and AB is α, and the angle between sides CB and AB is Let the angle between these be β. In this case, the coordinates (xc, yc) of point C are expressed by the following formula: xc = xa + L B cosα (12) yc=√{(LB 2 -(LBcosα) 2 )} ···(13) Here, cosα can be calculated according to the cosine theorem as follows: cosα = (LB 2 +L 2 -LA 2 ) / 2·LB·L···(14)

[0100] Here, when the surfaces of the two sign boards 211 and 212 are not aligned, as shown in FIG. In this case, the image captured by the camera 121 at point C is used to Then, the angles α1 and β1 with the sign surface are calculated using the above formulas (8) and (10). , and calculate the interior angles α and β. This allows the position information of point C to be calculated.

[0101] When using the function of calculating the focal length of the distance measuring sensor or camera 121, The terminal position may be calculated by photographing the identification plate 200. He explains.

[0102] As in the above embodiment, the angle between the sign surface and the mobile terminal 100 obtained from the captured image is defined as α1, The direction is assumed to be α2. Here, as shown in FIG. If the angle between the line segment and the east-west direction is α3, the coordinates (xc, yc) of point C can be expressed by the following formula. will be done. xc = xa + LBcosα3 (15) yc=ya-LBsinα3 (16) In addition, between α1, α2, and α3, α1+α3+(90 degrees - α2)=180 degrees ···(17) Therefore, α3=90°-α1+α2 (18)

[0103] <Modification 3 of the first embodiment> In addition, although the sign board 200 has been described above as an individually manufactured plate-like object, However, the present invention is not limited to this, and the image may be printed on a part of an advertisement inside a store. It is also possible to refresh the signboard 200 every time an advertisement changes and to add new information to the QR code. It becomes.

[0104] In addition, the location where the signboard 200 is installed does not need to be the same all the time, but can be moved. Each time the object moves, the position and direction information after the movement is added.

[0105] In the above description, it is assumed that the locations of the signboards 200 are not known in advance. However, this is not limited to these, and may be used at the entrances of underground shopping malls or large shopping centers. Also, each signboard 200 is posted in advance near each staircase, elevator, escalator, or in a place with a lot of foot traffic. You can display the location of the item, or you can register the information on a website. The signs are posted at the entrances of each store, at each product section, and in other places where customers can easily see and check them. It is definitely okay to install it.

[0106] <Fourth Modification of the First Embodiment> In the above embodiment, the calculated terminal position is displayed on the display 131. The processing of the calculation result by the calculation unit 172 is not limited to this. For example, The current position may be output to other applications that require it.

[0107] For example, in a large shopping center or home center, it is possible to locate a customer at the location of a desired store. It is an application that guides users to the

[0108] The application is stored in, for example, the ROM 111 or the storage 11 of the mobile terminal 100. 3. The CPU 101 loads it into the RAM and executes it according to the user's instructions. The functions executed by this application are described below as It is called navigation section 176.

[0109] This method will be explained with reference to Figs. 11(a) and 11(b). The store information is registered in advance and stored in the server 960. The information includes the store's description and location information. The location information is registered in latitude and longitude.

[0110] The navigation unit 176 is connected to a server 960 in which store information is registered via a network 940. The store information is then acquired by accessing the display 131 in a predetermined display format. Display it on.

[0111] When the user selects a desired store, navigation from the current location to the location of the store is started. The current location is calculated and displayed as the terminal location calculated by the above method. Use the position.

[0112] For example, in the example of FIG. 11(a), the navigation unit 176 displays the store on the map on the display 131. The stores are arranged and displayed according to the location information. The current location of the mobile terminal 100 is also displayed. The map information is stored in advance in the mobile terminal 100, for example, in the data storage unit 174. , or obtained from server 960.

[0113] In FIG. 11(a), the current location of the owner 910 of the mobile terminal 100 is in area A21. The east corner. The store selected by the owner is in the A13 area. 176 will display the route on this map.

[0114] At this time, as shown in FIG. 11(b), a navigation system is displayed to navigate from the current location to the desired store. The application information may be displayed in text form on the display 131. The target store is one block north and two blocks east. That's fine.

[0115] Moreover, according to this modification, it is possible to know in which direction the mobile terminal 100 is facing. Using the current location and direction information of the mobile terminal 100 and the location information of the target store, "Please go forward," "Please go backwards," etc., the direction in which the owner 910 is facing at that time. Navigation instructions may be provided based on the criteria.

[0116] In addition, each store will have detailed product information, and when you enter the store, the location of the product will be displayed. The product information may include a product name, a product description, and a product location. When the user selects a desired product, the system displays a list of product names, and then displays the route to that product in the store. Navigation may be configured to occur in a similar manner.

[0117] The navigation unit 176 transmits the display information to the access point 9 via the communication device 150, for example. 70 and from the server 960 via the network 940 .

[0118] <Fifth Modification of the First Embodiment> The location information used to calculate the location information of the mobile terminal 100 is limited to latitude and longitude. For example, each shopping center may use its own coordinate system. In this case, the position information of the above-mentioned product is also registered using coordinate values ​​in the same coordinate system.

[0119] In addition, QR codes for each store will be displayed at information centers in shopping centers, etc. Then, the mobile terminal 100 reads the QR code to navigate to the store. The device may be configured to perform the above-mentioned operations.

[0120] In addition, QR codes for products are displayed at the entrances of each store, and navigation is also available. may be carried out.

[0121] The location of the owner 910 changes from moment to moment while moving to the destination. Therefore, for the movement between the two points, for example, a three-axis calculation is required. The speed sensor 143, the three-axis gyro sensor 142, and the clock function are used to measure the distance and direction of movement. The detected and corrected value may be shown in the map information.

[0122] Furthermore, if it is determined that the error in the position estimation by the internal sensor has increased, The location of the signboard 200 is displayed to prompt the owner 910 to measure the location, or the mobile terminal 100 automatically For example, if the distance traveled from the previous location measurement point is large, If the vehicle becomes unstable or appears to be moving through a non-passageway on the map, It is determined that the error in the position estimation by the sensor has become large.

[0123] <<Second embodiment>> Next, a second embodiment of the present invention will be described. In the first embodiment, a sign board 200 is provided with a sign The ratio of distortion of the sign shape in the horizontal plane and the direction perpendicular to it in the recognition image is used to determine the two The length of the line segment connecting the center points of the signboard 200 and the angles at both ends are calculated, and the terminal position is calculated. There are.

[0124] In this embodiment, a circle is drawn on the sign board 200, and the distortion of the circle in the captured image is detected. From the degree, information necessary for terminal position estimation is obtained. Unlike the embodiment, the sign board 200 and the mobile terminal 100 do not have to be on the same horizontal plane. Also, only one sign board 200 may be used for capturing an image.

[0125] In addition, in this embodiment and the third embodiment, unlike the first embodiment, the position information is a height It is a three-dimensional quantity including the two lines that are not on the same line and whose positions are known. Consider at least two triangles formed by each line segment and the terminal position, and The device position is estimated in three dimensions by calculating the angles and lengths of each side of the two triangles. Since the shapes share the terminal position, the surface orientation of the triangles is determined by calculating the shape of each triangle. This makes it possible to calculate the terminal position.

[0126] Hereinafter, the present embodiment will be described, focusing on the configuration different from the first embodiment.

[0127] [Sign board] Before describing the method for calculating the terminal position, the signboard 230 of this embodiment will be described. As shown in FIG. 12, the sign board 230 of this embodiment has a circular sign circle 231. , a direction line 232, and a position and direction information display area 233.

[0128] The position and direction information display area 233 displays the information on the terminal position calculated by photographing the signboard 230. For example, the position of the center of the marker circle 231 is recorded. The information, the diameter of the marker circle 231, the bearing line direction, and the surface normal direction are recorded.

[0129] The center position information is registered as, for example, latitude, longitude, and altitude. 32 three-dimensional directions, for example, represented by the components of a unit vector. The three-dimensional direction of the normal of the marking surface of the identification plate 230, for example, is represented by the components of the surface normal vector. can be.

[0130] In addition, the azimuth line 232 may be marked with, for example, an arrow, a triangle, or the like, so that the direction can be easily understood. It may be possible.

[0131] [Mobile device] The hardware configuration and functional blocks of the mobile terminal 100 of this embodiment are the same as those of the first embodiment. They basically have the same configuration, except that the calculation method used by the calculation unit 172 is different.

[0132] [Position calculation method] Next, the calculation unit 172 captures an image of the signboard 230 and calculates the current position of the mobile terminal 100. The method will be explained.

[0133] In the following, in this embodiment, the distance in real space between point I and point J is L IJ Also, At points I, J, and K, which are not on a line, the angle between line segment IJ and line segment JK is called angle I. Let's call this angle JK. Also, let's call this angle IJK the angle from point J to line segment IK, or the angle from point J to line segment IK. It is also called the angle between points I and K. A and V B About Then, the vector product is [V A , V B ], the scalar product (V A , V B ) is expressed as

[0134] The point on the sign circle 231 of the sign board 230 used to calculate the terminal position is called a sign point. The coordinates of the spotting point can be calculated from the position and direction information written on the signboard 230. The angle of view between the two landmarks from the terminal position is calculated by multiplying the focal length of the camera 121 by 2. The geometry is calculated from the distance between the landmarks on the image sensor obtained from an image with two landmarks in the field of view. The focal length can be calculated based on the positional relationship between the lens of the camera 121 and the image sensor. Then, the remaining points in the triangle formed by the two landmarks and the terminal position are identified. The known length of the side and the value of the interior angle are calculated from the coordinate values ​​of the landmarks and the terminal positions of the two landmarks. The terminal position is calculated from the measured angle of view.

[0135] Furthermore, if the distance sensor can be used while taking into account the measurement error, the output of the distance sensor can also be It may be used.

[0136] In FIG. 13(a), a person standing at a position away from the location where the single signboard 230 is installed, such as a pillar, 9 shows a state in which an owner 910 is taking a picture using the mobile terminal 100.

[0137] The location (terminal location) of the mobile terminal 100 (owner 910) is X (point X). The person 910 is standing at an angle to the surface of the sign board 230 (in a direction different from the normal direction of the sign board). In this case, the display of the mobile terminal 100 is In the ray 131, the sign circle 231 of the sign board 230 is the same as the image of the sign circle 231 shown in FIG. The marker circle image 231i is displayed in an elliptical shape.

[0138] In addition, it is not limited that the mobile terminal 100 and the signboard 230 are on the same horizontal plane. Therefore, the mobile terminal 100 and the sign board 230 may have an angle in the height direction. The major axis 234i of the ellipse in 31i may be different from the orientation line on the image (orientation line image 232i). Calculations are made under these assumptions.

[0139] FIG. 13(c) shows the marker circle image 231i displayed on the display 131 of FIG. 13(b). In the following explanatory diagrams, the indicator circle 231 is used for the sake of simplicity. Images other than those corresponding to the azimuth line 232 are omitted.

[0140] The center point of the marker circle image 231i is Oi. Also, points Ai and Bi are respectively The intersection points Ci and Di between the orientation line image 232i and the marker circle image 231i are the major axis The intersection points Ei and Fi of the ellipse of the marker circle image 234i and the marker circle image 231i are the minor axis and the marker circle, respectively. This is the intersection point with the ellipse in image 231i.

[0141] In addition, the calculation unit 172 analyzes the captured marker circle image 231i to obtain Each point Ai to Fi is found, and the lengths of the major and minor axes on the image are calculated.

[0142] Next, the image is stretched in the direction of the minor axis on the image using the ratio of the lengths of the major axis and minor axis on the image described above. FIG. 14(a) shows the result of the transformation so that the circle 231 overlaps with the marker circle 231. The actual sign circle 231 and the azimuth line 232 are transformed by image expansion to the points Ai and B i, point Ci, point Di, point Ei, and point Fi are denoted as A, B, C, D, E, and F, respectively. So, let angle AOC be ηAC It is written as follows.

[0143] Here, the length of each line segment AB, CD, and EF is known as the diameter of the marked circle 231, and the line segment AB Since the sign 230 is a azimuth line 232, its three-dimensional direction is known. The normal direction is also known. Using these, the three-dimensional direction of the line segment CD and the three-dimensional direction of the short axis EF are Obtain the spatial coordinates of the axis end points (C, D, E, F) to be used as direction and landmark points. The center point O of the circle 231 is also used as a landmark, but the spatial coordinates of O are known from the position and orientation information. be.

[0144] Consider two or more triangles to calculate the terminal position X. First, as shown in Figure 14(b), Consider a triangle XOC with terminal position X, point C, and point O as vertices (first triangle). (It may be a triangle XOD). In the marker circle image 231i, a line segment CD (first diameter) is Since the terminal position X is in the direction of the major axis, it is possible to determine the position 236 perpendicular to the line segment CD. exists.

[0145] That is, the line segment CD is a straight line ( Therefore, the triangle XOC is a right-angled triangle, as shown in Figure 15(a). Also, the distance L between point O and point C OC is the radius R (1 / 2 the diameter) of the marked circle 231. The diameter is obtained by analyzing the area corresponding to the position and orientation information display area 233 on the captured image. Therefore, the distance L between point X and point O is OX can be calculated using the following formula. , the angle CXO, θ C Let θ C is measured from the image captured by the camera 121. L OX = R cotθC (19)

[0146] Next, to calculate the angle that the triangle XOC makes with the landmark, place it on the same line as the line CD. Consider a triangle (second triangle) formed by two landmarks that are not included in the triangle and the terminal position X. Let us use the line segment EF as the second diameter, and define the triangles XOE and XOE in Figure 15(b). Consider the shape XOF.

[0147] As shown in Figure 15(b), the angle EXO is θ E , angle FXO is θ F , angle EOX is λ E ,angle F OX to λ F , the angle OEX is ξ E , angle OFX is ξ F , let us assume that. By the sine law, the following equation holds: L OX / sinξ E =L OE / sinθ E (20) L OX / sinξ F =L OF / sinθ F ···(twenty one)

[0148] Here, L OE =L OF However, here we use L OE =L OF Without using the relationship For the sake of simplicity, the following notation is used: E and κ F is used. κ E =L OE / sinθ E ···(twenty two) κ F =L OF / sinθ F ···(twenty three)

[0149] L OE , L OF is the radius R of the marker circle 231, which is known. E , θ F is calculated geometrically from the focal length of the camera 121 and the length on the image sensor corresponding to EF. The focal length can be calculated based on the positional relationship between the lens and the image sensor in the camera 121. Therefore, κ E , κ F can be calculated using these.

[0150] κ E , κ F Using this, the distance L between the terminal position X and the signboard 230 is OX is expressed by the following formula: . L OX =κ E sinξ E =κ F sinξ F ···(twenty four)

[0151] Here, we want to find λ E Therefore, the above relation can be expressed as λ E If we rewrite it using It is expressed by the formula: ξ E =180 degrees-λ E -θ E ···(twenty five) ξ F = λ E -θ F (26) Therefore, the distance L OX is expressed by the following formula: L OX =κ E sin(λ E +θ E )=κ F sin(λ E -θ F ) ···(27)

[0152] By expanding and rearranging the above formula, the following formula is obtained according to the addition theorem. sin λ E (κ F cosθ F -κ E cosθ E )=cosλ E (κ E sinθ E +κ F sinθ F ) ···(28)

[0153] Here, sin λ E , and (κ E sinθ E +κ F sinθ F ) is positive, so cos λ E and (κ F cosθ F -κ E cosθ E ) and the signs are the same. Taking the square of both sides of the above equation, we get the following equation: cos 2 λ E {(κ E sinθ E +κ F sinθ F ) 2 +(κ F cosθ F -κ E co sθ E ) 2}=(κ F cosθ F -κ E cosθ E ) 2 (29) In this case, sin 2 λ E +cos 2 λ E =1 relationship was used.

[0154] cos λ E and (κ F cosθ F -κ E cosθ E ), if we pay attention to the sign relationship with cos λ E is calculated as follows: cos λ E =(κ F cosθ F -κ E cosθ E ) / {(κ E sinθ E +κ F sin θ F ) 2 +(κ F cosθ F -κ E cosθ E ) 2} 1 / 2 (30) Similarly, sin λ E is calculated as follows: sin λ E =(κ E sinθ E +κ F sinθ F ) / {(κ E sinθ E +κ F sin θ F ) 2 +(κ F cosθ F -κ E cosθ E ) 2} 1 / 2 (31)

[0155] κ E ,κ F If we revert the notation, we get the following: cos λ E =(L OF cotθ F -L OE cotθ E ) / {(L OE +L OF ) 2 +( L OF cotθ F -L OE cotθ E ) 2} 1 / 2(32) sin λ E =(L OE +L OF ) / {(L OE +L OF ) 2 +(L OF cotθ F -L OE cotθ E ) 2} 1 / 2 (33)

[0156] Therefore, L OX is expressed by the following formula: L OX =κ E sinξ E =(L OE / sinθ E )sin(λ E +θ E ) =(L OE / sinθ E )(sinλ E cosθ E +cosλ E sinθ E ) =L OE L OF (cotθ E +cotθ F ) / {(L OE +L OF ) 2 +(L OF cotθ F -L OE cotθ E ) 2} 1 / 2 (34) Since it is not used in the following calculations, the details are not shown, but the L of the remaining sides EX and L FX Also sine constant That is, by the above method, all the triangles formed by E, O, F, and X can be found. The angles and sides of can be calculated.

[0157] Here, LOE =L OF The above equation can be rewritten using the =R relationship as follows: cos λ E =(cotθ F -cotθ E ) / {4+(cotθ F -cotθ E ) 2} 1 / 2 (35) sin λ E =2 / {4+(cotθ F -cotθ E ) 2} 1 / 2 (36) Therefore, L OX is calculated using the following formula: L OX =R(cotθ E +cotθ F ) / {4+(cotθ F -cotθ E ) 2} 1 / 2 (37)

[0158] As shown in Figure 15(b), three points on the same line with known inter-point distances and the distance from terminal position X are From the measurement of two angles between two of the three points, the terminal position X and the angle formed by the three points are The unknown angles and lengths of the sides of the triangle can be calculated. That is, the terminal position X is As shown in FIG. 15(c), the distance L from the line segment CD OX Of the 236 positions, the angle EOX is λ E is identified as the position of

[0159] In addition, the distance L OX As described above, the point corresponding to the major axis of the marker circle image 231i Using point C and point D, as shown in the above formula (19), L OX =Rcotθ C It can be calculated as . distance L OX Either method may be used to calculate the value, or the average of the two may be used. stomach.

[0160] Next, the terminal position X is calculated. Here, the central angle AOC of the arc AC shown in FIG. η AC For , the counterclockwise direction from point A to point C is considered positive.

[0161] FIG. 16(b) shows the direction of each unit vector, which will be described later. Let V be the unit vector from point O to point A in the distance. A Then, the sign in the real space The surface normal vector (unit vector) of 230 is V S And V A Clockwise around O A vector rotated 90 degrees in the plane of the label is V T Let us assume that.

[0162] V T is V A and V S Since it is calculated as a vector product of It can be calculated. V T =[V A ,V S ] ···(38) Here, the vector V T The angle between and the line segment OE is η AC is equal to.

[0163] Let V be the unit vector from point O to point X. X Let us assume that V X is perpendicular to the long axis CD For this reason, V X The projection of onto the plane of the sign is on the line EF. Also, V X and the line segment OE The angle is λ E Therefore, V X is calculated using the following formula: VX =cosλ E cosη AC V T +cosλ E sinη AC V A +sinλ E V S (39)

[0164] The position coordinates of the center point O of the sign board 230 in the real space are P O , the location of the terminal position X in the real space Mark P X Then, P X is calculated using the following formula: P X =P O +L OX V X (40)

[0165] As described above, in this embodiment, the position information of the center point and the orientation information of the surface are known. In addition, the marker circle 231 on the sign board 230 and the chord passing through the center of the marker circle 231 are The sign 230 having the azimuth line 232 is photographed, and the photographed image is analyzed to identify the sign. Two predetermined points where a predetermined diameter and a circumference on the identification circle 231 intersect, and a baseline connecting the two points and a mobile terminal The two end angles connecting 100 and two specified points are calculated, and the terminal position is calculated. The signboard 230 analyzes the captured image to obtain the above two pieces of location information and the terminal location. The present invention is also configured to be able to calculate the direction of a plane including the above two points.

[0166] In this way, according to this embodiment, like the first embodiment, the sign board 230 can be photographed. Therefore, according to this embodiment, the terminal position can be calculated in the same manner as in the first embodiment. Even if the GPS function cannot be used, the mobile terminal 100 can be accurately located with a simple configuration. The current location and heading of the owner 910 can be calculated.

[0167] Furthermore, according to this embodiment, the terminal position X is obtained as a three-dimensional position including the height. Moreover, the direction from the terminal position X to the sign board 230 is obtained as a direction vector in real space. Therefore, it is also possible to know in which direction the body of the mobile terminal 100 is facing in real space. This allows precise calibration including the three-dimensional directions of the internal coordinate system of the mobile terminal 100, so the height can be It is also possible to guide the location of the target point by superimposing the image on the actual image. It is also possible to display images that

[0168] For example, in the location of a shelf where a product is stored, the product position can be superimposed on the image of the shelf. It may also be displayed as an Augmented Reality (R) image. An example is shown in FIG. 7(a). The display 131 of the mobile terminal 100 displays the image captured by the mobile terminal 100. The system displays a real-life image 301 of a product shelf and an AR image 300 showing the location of the product. Detailed location information, including product height, is obtained from the store via the Internet. Also, the actual product shelf image 301 may be obtained from the store after being shot in advance. In this case, the owner 910 is looking at the inside of the store from the direction in which the mobile terminal 100 is facing. Alternatively, the part may be estimated and the image may be displayed.

[0169] In addition, the navigation screen is displayed in a flat manner as shown in Fig. 11(a) until the user arrives near the target product. The location of the owner 910 is indicated on the map, and when the owner arrives near the target product, Automatically switches to a mode that displays actual images of the product shelves, and displays images of the product shelves and the target product. In this case, if the target is within the field of view of the camera 121 of the mobile terminal 100, If the product image is not included in the product list, which direction should the mobile terminal 100 be pointed? (Fig. 17(b)). The item is in a visible position, and the owner 910 is within a predetermined distance from the item. You can decide based on whether the conditions are met.

[0170] Furthermore, the goal of location guidance is not limited to products, but also to the process of reaching a destination. It may also indicate a route, such as a door.

[0171] In particular, according to this embodiment, the current position and direction can be obtained by simply photographing one signboard 230. In addition, when the mobile terminal 100 is not on the same horizontal plane as the signboard 230, the position of the mobile terminal 100 can be calculated. The terminal position can be calculated by a simpler configuration than the first embodiment. In addition, the degree of freedom in installing the signboard 230 is increased, and it can be installed on the ceiling or floor, for example. This allows the owner 910 of the mobile terminal 100 to see the sign 230 more clearly. This increases the possibilities for calculating the terminal position.

[0172] <<Third embodiment>> Next, a third embodiment of the present invention will be described. In this embodiment, the shape of the marker is not used. Using multiple landmarks, each of which has a known three-dimensional position, the position of the mobile terminal 100 (terminal position Calculate the position.

[0173] Hereinafter, the configuration of the mobile terminal 100 of this embodiment is basically the same as that of the first embodiment. The following description will focus on the configuration different from the first embodiment. is the same as in the second embodiment.

[0174] As described above, in this embodiment, the shape of the sign is not used to calculate the terminal position. The three-dimensional position information of the landmarks is used. Here, the terminal position is calculated using four landmarks. The four landmarks are to be on the same plane. This section explains an example where a landmark is indicated on a single signboard.

[0175] [marker point] An example of the landmark of this embodiment is shown in FIG. 18. As shown in this figure, in this embodiment, one Four marking points 241 are arranged on the sign board 240. In this embodiment, the sign board 240 Assume that four points are placed at each vertex of the imaginary quadrangle above. That is, three or more points are the same. The landmarks 241 are not arranged on a straight line. Hereinafter, the landmarks 241 are referred to as points A, B, and C. , called point D.

[0176] In addition, if there are landmarks 241 on each of the four vertices of a quadrangle, the intersection of the diagonals 242 The point O may be displayed on the signboard 240 or may be stored as internal data for image processing. It may be treated as data.

[0177] The signboard 240 of this embodiment includes a position information display area 243. In the field 43, the position coordinates of each landmark 241 are described. The latitude, longitude, and altitude of the landmark 241. The position coordinates of the intersection 242 (point O) are 241, or may be written in the position information display area 243. stomach.

[0178] In this embodiment, the image acquisition unit 171 acquires four landmarks 241 and the positions of the landmarks 241. The information is acquired as an image.

[0179] Then, the calculation unit 172 analyzes the image acquired by the image acquisition unit 171 and calculates the image of the mobile terminal 100. The location information of the terminal position X is calculated as the current location.

[0180] An example of a method for calculating the current position by the calculation unit 172 of this embodiment will be described below. As shown in FIG. 18, the four landmarks 241 are a line segment connecting point A and point C, and a line segment connecting point B and point C. It is assumed that the line segment connecting D and θ intersects at point O.

[0181] In order to estimate the terminal position X, as shown in Figs. 19(a) and 19(b), , 241 marked points are the group AO, OC, and the group BO, OD. Consider the following. The lines between the groups are not aligned. The shape of these lines and the terminal position X Calculate the angles and sides of the triangle. Here, the distance between each landmark 241, including point O, is calculated. distance L OA , L OC ,L OB , L OD can be calculated using the position information. Also, the angle AXO (θ A ), angle CXO(θ C ), angle BXO(θ B ), and angle DXO(θ D ) are respectively As described with reference to FIG. 15(b), the focal length of the camera 121 and the corresponding image sensor The distance can be calculated geometrically from the length of the sensor using the captured image.

[0182] Hereinafter, as shown in Fig. 19(a) and Fig. 19(b), the angle XAO is expressed as ξ A , the angle XCO is ξ C , the angle XOA is λ A , the angle XOC is λ C , the angle XBO is ξ B , the angle XDO is ξ D , the angle XOB is λ B , the angle XOD is λ D and respectively.

[0183] Using the same calculation as in the modified example of the second embodiment described with reference to FIG. A ,λ B , L OX can be calculated by the following formulas, respectively. cos λ A =(L OC cotθ C -L OA cotθ A ) / {(L OA +L OC ) 2 +( L OC cotθ C -L OA cotθ A ) 2} 1 / 2 (41) sin λ A =(L OA +L OC ) / {(L OA +L OC ) 2 +(L OC cotθ C -L OA cotθ A ) 2} 1 / 2 (42) cos λ B =(L OD cotθ D -L OB cotθ B ) / {(L OB +L OD ) 2 +( L OD cotθ D -L OB cotθ B ) 2} 1 / 2 (43) sin λ B =(L OB +L OD ) / {(LOB +L OD ) 2 +(L OD cotθ D -L OB cotθ B ) 2} 1 / 2 (44) L OX =L OA L OC (cotθ A +cotθ C ) / {(L OA +L OC ) 2 +(L O C cotθ C -L OA cotθ A ) 2} 1 / 2 (45) L OX =L OB L OD (cotθ B +cotθ D ) / {(L OB +L OD ) 2 +(L O D cotθ D -L OB cotθ B ) 2} 1 / 2 (46)

[0184] As shown in Fig. 19(c), λ A ,λ B ,L OX Once is determined, the terminal position X is uniquely determined. do.

[0185] In this embodiment, the distance L OX is the triangle AXC or triangle BX as described above. D. As in the modified example of the second embodiment, the calculation can be performed by using only one of the values. For example, the calculation unit 172 may calculate the expected distance from the terminal position X to the line segment AC. Angle (θ A +θ C ) and the angle of view of the line segment BD (θ B +θ D ) and compare the two Then, using the larger one, distance L OX Alternatively, the average of the two can be used to find Good too.

[0186] Next, the terminal position X is calculated. In this embodiment, as shown in FIG. Let V be the unit vector pointing towards point A. A Let us assume that the unit vector in the direction from the intersection point O to point B is V B Both can be calculated from the coordinates of the landmarks. Let V be the unit vector perpendicular to the surface and pointing towards the viewer of the sign. S Then, move counterclockwise from point B to point A. The angle BOA measured by η BA Let us assume that V S is calculated using the following formula: V S =[V B ,V A ] / sinη BA (47)

[0187] Furthermore, let V be the unit vector going from the intersection point O to point X. X Furthermore, as shown in FIG. As you can see, V X The angle (V X and V S The complement of the angle between X (≦90 degrees). Now, by applying the vector triple product formula, V X ,V B ,V A The relationship between is expressed by the following equation: [V X ,[V B ,VA ]]=(V X ,V A )V B -(V X ,V B )V A (48 ) Take the square of both sides of the above equation (as an inner product). In this case, V B ,V A is expressed by the above formula (47). From V S That is, V X ,V B ,V A The relationship between It is expressed by the formula: [V X ,[V B ,V A ]]=[V X ,V S ]sinη BA (49) Taking the square of both sides of the above equation (as an inner product), the left and right sides become the following equations, It is expressed as: (Left side) 2 =sin 2 η BA ([V X ,V s ],[V X ,V s ]) =sin 2 η BA sin 2 (90 degrees-λ X ) =sin 2 η BA cos 2 λ X ···(50) (right side) 2 =((cosλ A V B -cosλ B V A ),(cosλ A V B -cosλ B V A )) =cos2 λ A +cos 2 λ B -2cosλ A cos λ B cosη BA · (51) Therefore, from the above equation, cosλ X can be calculated using the following formula: cos λ X =(cos 2 λ A +cos 2 λ B -2cosλ A cos λ B cosη BA ) 1 / 2 / |sinη BA | (52)

[0188] Here, cosλ X If = 0, then λ X = 90 degrees, i.e., V X =V S It is.

[0189] Next, cosλ X ≠0(λ X <90 degrees).

[0190] First, as shown in FIG. 20(b), the intersection of the perpendicular line from the terminal position X to the sign surface with the sign surface is calculated. Let point H be the unit vector from point O to point H. H Here, the following vector Torr product [V X ,V S ] is a unit vector in the direction of V T Then, V T is expressed by the following formula: do. V T =[V X ,V S ] / cosλ X =[V X ,[V B ,V A ]] / (cosλX sinη BA ) ={(V X ,V A )V B -(V X ,V B )V A} / (cosλ X sinη BA ) =(cosλ A V B -cosλ B V A ) / (cosλ X sinη BA ) ···(5 3) V T is parallel to the sign surface, and V T and V H are orthogonal, and V T Rotate it 90 degrees counterclockwise V H Therefore, V H is calculated as follows: V H =[V S ,V T ] =[V S ,(cosλ A V B -cosλ B V A )] / (cosλ X sinη BA ) =[[V B ,V A ],(cosλ A V B -cosλ B V A )] / (cosλ X sin 2 η BA ) = {cosλ A [[V B ,V A ],V B] -cosλ B [[V B ,V A ],V A ]} / (cosλ X sin 2 η BA ) = {cosλ A (V A -cosη BA V B )-cosλ B (cos BA V A -V B )} / (cosλ X sin 2 η BA ) = {(cosλ A -cosλ B cosη BA )V A +(cosλ B -cosλ A co sη BA )V B} / (cosλ X sin 2 η BA ) ···(54)

[0191] From the above, the position coordinate of the intersection point O is P O (obtained from sign information), the location coordinate of terminal position X P X Then, P X is calculated as follows: P X =P O +L OX sin λ X V S +L OX cos λ X V H =P O +(L OX / sin 2 η BA )×{sgn(sinη BA )(sin2η BA -cos 2 λ A -cos 2 λ B +2cosλ A cos λ B cosηBA ) 1 / 2 [V B , V A ]+(cosλ A -cosλ B cosη BA )V A +(cosλ B -cosλ A co sη BA )V B} ···(55) Here, sgn(sinη BA ) is as follows: sgn(sinη BA )=+1 if sinη BA >0 (56) sgn(sinη BA )=-1 if sinη BA <0 ···(57)

[0192] cos λ X When V = 0, that is, X When is perpendicular to the label plane, P X =P O +L OX V S cos λ A =cosλ B = 0, then the above equation becomes . P X =P O +L OX [V B ,V A ] / sinη BA =P O +L OX V S ···(58 ) In the above formula, cosλ X This includes the case where =0.

[0193] As described above, in this embodiment, four landmarks 241 on the same plane are photographed, and the images are The terminal position of the mobile terminal 100 is calculated by image analysis. For example, four The line segment connecting two landmarks (point A and point C) of landmark 241 and the terminal position (point X) The angles (angles XAC and XCA) formed by the line segment connecting the two landmarks A and C, and the point X, The direction of the plane including A and C is calculated. Estimate the terminal location.

[0194] That is, according to this embodiment, the terminal position is calculated by photographing four different landmarks. Therefore, according to this embodiment, as in the above embodiments, even in cases where the GPS function cannot be used, Even in such a case, the current location and The direction can be calculated.

[0195] In particular, according to this embodiment, when one sign board 240 having four sign points 241 is photographed, The current position and direction can be calculated simply by calculating the terminal position X. The direction from the terminal position X to the signboard 240 is calculated as a 3-dimensional position. Therefore, the direction in which the body of the mobile terminal 100 is facing in the real space is This makes it possible to guide the location, including the height.

[0196] That is, according to this embodiment, the mobile terminal 100 is on the same horizontal plane as the signboard 240. Therefore, the degree of freedom in attaching the signboard 240 is increased. At the same time, the possibilities for calculation are expanded for the owner 910 of the mobile terminal 100 as well.

[0197] <Modification 1 of the third embodiment> In this embodiment, the position information does not have to be provided on the signboard 240. It is on the same imaginary plane, and four landmarks 241 are photographed at the same time, and their information can be obtained. If the landmarks 241 are located at distant positions, the landmarks The position coordinates of each point may be recorded.

[0198] In the following modified examples, including this modified example, all landmarks 241 do not fit within one image. In that case, the measurement of the angle of view between the two landmarks from the terminal position is 3 The output of the axis gyro sensor 142 is used.

[0199] For example, suppose landmark A is photographed in image a, and landmark B is photographed in image b. In each image, the distance between the terminal position and the center of the image and landmark A in the image, The angle of view between each of the landmarks B and the camera 121 is calculated by the focal length of the camera 121 and the landmark B. The distance between the points on the image sensor can be calculated geometrically. The spatial orientation in the internal coordinate system is calculated from the output of the three-axis gyro sensor 142. The directions of landmarks A and B in the spatial direction of the internal coordinate system can be calculated, and the two landmarks can be calculated from the terminal position. The angle of view between A and B can be calculated. Here, in calculating this angle, Since the difference in the center direction between image a and image b only needs to be determined relatively, the three-axis gyro sensor 142 The orientation of the internal coordinate system calculated from the output may deviate from the orientation in real space.

[0200] In the present embodiment, location information such as a URL is also displayed on the signboard 240. The location of each landmark 241 is obtained from the management server via the QR code. The present invention may be configured to obtain the above-mentioned.

[0201] In addition, the position coordinates do not have to be latitude, longitude, and altitude. For example, the sign 240 may be located in a shopping mall. If the information is located in a shopping center, it will be linked to the floor map of the shopping center. It may be possible to do so.

[0202] <Modification 2 of the third embodiment> Also, each of the mark points 241 does not have to be a vertex of a rectangle. It is sufficient that no single point is on a line connecting two or more other landmark points 241.

[0203] Of these, three landmarks 241 are on a straight line, and one landmark 24 is not on the straight line. The method of calculating the current position by the calculation unit 172 in the case of the sign board 240 having the number 1 is shown in FIG. ) to FIG. 21(c).

[0204] In this modification, as shown in FIG. 21(a), points A, O, and C are on a straight line, and point B The coordinates of each landmark 241 are written in the location information display area 243. The configuration other than the arrangement of the landmarks 241 is the same as in this embodiment.

[0205] FIG. 21(b) shows a triangle AXC that connects the terminal position X, point A, and point C. Similarly to the embodiment, the angle AXO is θ A , the angle CXO is θ C , the angle XAO is ξ A , the angle XCO is ξ C , angle X OA to λ A , the angle XOC is λ C , and , respectively. This triangle AXC is shown in Figure 19(a). This is the same structure as the triangle AXC in the above equation. Therefore, by the same calculation as above, L OX and λ E is required .

[0206] FIG. 21(c) shows a triangle BXO that connects the terminal position X, point O, and point B. Similarly to the embodiment, the angle BXO is θ B , the angle XBO is ξ B , the angle XOB is λ B , and , respectively.

[0207] L OB The value of is calculated from the position information. B is obtained by measurement using camera 121. And L OX can be calculated from the properties of triangle AXC. Therefore, the sine of the following equation By the theorem, ξ B is obtained, and λ B is also required. L OB / sinθ B =L OX / sinξ B (59)

[0208] Thereafter, the terminal position X is found by the same calculation as that explained with reference to FIG.

[0209] As described above, according to this modified example, the points are on the same plane and at least one point is on the same straight line. If the position information (position coordinates) of the four landmarks 241 not on the map is given, the mobile terminal 100 By measuring the angle of view between each landmark from the terminal position X, the mobile terminal 10 It is possible to calculate the terminal position X of 0. In addition, the same effects as those of the above embodiment can be obtained.

[0210] <Modification 3 of the third embodiment> (When the four landmarks are not on the same plane) In addition, the four landmarks 241 do not have to be on the same plane. As long as the terminals are not arranged in a straight line, the terminal position X can be calculated. The method of calculating the terminal position X by 72 will be described below.

[0211] First, divide the four landmarks A, B, C, and D into pairs of two landmarks each. Consider two lines passing through the points. Here, we consider the pair of points A and B and the pair of points C and D. In addition, as shown in FIG. 22(a), the classification is performed by dividing the points A and B into two groups. Select a line AB and a line CD passing through points C and D so that they are not parallel.

[0212] FIG. 22(a) is a diagram showing each landmark 241 viewed from the direction of the terminal position X. Figures 22(b), 22(c), and 22(d) show the planes determined by the terminal position X and the line segment CD. 2 is a diagram showing each landmark 241 viewed from the normal direction of the map.

[0213] When viewed from the mobile terminal 100, two points on a line in the same direction (or in opposite directions) are The intersections on the intersections are called new additional landmarks O and U (hereinafter simply called points O and U). In addition, the "opposite direction" refers to a direction in which the mobile terminal 10 is in a straight line with the mobile terminal 10 as shown in FIG. This is the case where 0 is sandwiched between the two points. Point O is a point on the line AB, and point U is a point on the line CD. Here, In this case, point O is closer to the terminal than point U. As shown in FIG. 22(d), point O And point U is not necessarily between points A and B, or between points C and D.

[0214] Here, we consider a plane determined by the terminal position X and the line AB. On this plane, there are points O and and point U also exist.

[0215] An example of the relationship between landmarks A, B, and U and the terminal position X of the mobile terminal 100 on this plane is shown in FIG. The intersection point of the line segment AB and the line segment UX is O. The angles of each vertex are as shown in the figure. Then, let the angle UAO be η A , the angle UBO is η B , the angle AUO is ζ A , the angle OUB is ζ B ,corner XAO A , the angle XBO is ξ B , the angle AOX is λ A , the square box is λ B , the angle AXO is θ A ,corner BXO θ B , as expressed.

[0216] The following equation holds from the sine theorem for triangles XOA and XOB. L OX =L OA sinξ A / sinθ A =L OB sinξ B / sinθ B ...(6 0) Additionally, the following equation holds from the sine theorem for triangles UOA and UOB: L OU =L OA sinη A / sinζ A =L OB sinη B / sinζ B ...(6 1) From the above two equations, the following relationship holds: sinξ A sinζ A / sinθ A sinη A =sinξ B sinζ B / sinθ B s inη B (62) This relational expression can be transformed into the following expression. cot A ( A +cot B -cotθ A -cotθ B)=cotθ B cot η B -cotθ A Cote A (63)

[0217] Once the location of the landmark U is determined, η A and η B is determined, and from the above equation, λ A Therefore, Fig. The interior angles and side lengths of each triangle shown in 23(a) are calculated, and the triangle XA in which the terminal position X is located is Since the orientation of face B is also determined, the terminal position X can be found.

[0218] Here, λ A There are exceptional cases where the interior angles are not determined. This is the case when the relationship shown in the formula is satisfied. Cote A +cot B -cotθ A -cotθ B =0 (64) In this case, the following relationship also holds: cotθ B Cote B -cotθ A Cote A =0 (65) By solving the above two equations simultaneously and transforming them, we obtain the following equation. Cote A =cotθ B (66) Cote B =cotθ A (67) That is, η A , η B , θ A , θ B are the interior angles of a triangle, so The relationship is established. η A =θ B (68) η B =θ A (69)

[0219] Considering the inscribed angle theorem, it means that points A, B, U, and X are on the same circumference. On the other hand, point X is any position on the circumference where the above formula holds. Therefore, λ A Is fixed In this case, move the mobile terminal 100 to another location and repeat the measurement. The calculation unit 172 then outputs a message prompting the user to move and re-shoot.

[0220] Except for exceptional cases, once the landmark U is determined, the terminal position X can be determined. The landmark U is a line Therefore, when calculating the terminal position X by assuming that each point on the line CD is a landmark U, In this case, the candidate points for the terminal position X lie on a predetermined curve (first curve) in space.

[0221] As shown in FIG. 23(b), a similar calculation is performed using landmarks C, D, O, and U. Fruit, λ C is calculated using the following formula: cot C ( C +cot D +cotθ C +cotθ D )=cotθ D cot η D -cotθ C Cote C ···(70)

[0222] Similarly, if we calculate the terminal position X by assuming that each point on the line AB is point O, the terminal position A curve (second curve) different from the first curve is determined on which the candidate point of X is placed. The terminal position X is the first The intersection of the first curve and the second curve.

[0223] As shown in Figure 24(a), in triangle ABU, drop a perpendicular line from point U to side AB, Let the foot of the perpendicular be W. A, cot B Expressing this in terms of inter-point distance and rearranging it gives us equation (63 ) is expressed as follows: cot A (L AB -L UW (cotθ A +cotθ B ))=L BW cotθ B -L A W cotθ A ···(71)

[0224] Next, in triangle CDO, draw a perpendicular line from point O to side CD as shown in Figure 24(b). However, if the foot of the perpendicular line is Q, equation (70) can be expressed as follows. cot C (L CD +L OQ (cotθ C +cotθ D ))=L DQ cotθ D -L C Q cotθ C (72)

[0225] For example, as shown in Figure 18, when landmarks A, B, C, and D are on the same plane, 1) and (72), the points U, O, W, and Q are the same point, and the limit of (4 1) does not contradict the notation in equation (44).

[0226] Here, the lengths (distances) of the lines AO, OX, CU, and UX are calculated as follows: will be done. L AO =L AW -L UW cot A ···(73) L OX =(L AW -L UW cot A )(cotθA +cotλ A )sinλ A ·· (74) L CU =L CQ +L OQ cot C (75) L UX =(L CQ +L OQ cot C )(cotθ C +cotλ C )sinλ C ·· (76)

[0227] By determining point U, point W is also determined, and the terminal position is calculated using equations (71), (73), and (74). The coordinate of X is determined. On the other hand, when point O is determined, point Q is also determined, and equations (72), (75), and ( 76) is used to determine the coordinates of the terminal position X. The terminal position X calculated by both is the same coordinate. Search for points U and O so that

[0228] As described above, the coordinate values ​​of the four landmarks 241 in the external coordinate system are known. Given a landmark 241 that is not on a plane and has at least one point that is not on the same line, Measure the angle of view from the terminal position X between the sign point 241 and the intersection point, and compare the signs in different directions. The angle between the line connecting the points 241 and the line connecting the landmark 241 and the terminal position X, and the landmark Calculate the distance from point 241 to terminal position X, and calculate the coordinate value of terminal position X in the external coordinate system. The intersection point can be determined by looking from the terminal position X of the two straight lines connecting the landmarks 241. It is either the apparent intersection point (point O, point U). Also, there are two or more signs from the terminal position X. Since the direction in the external coordinate system toward the point 241 is known, the external coordinate system of the internal coordinate system of the mobile terminal 100 is The orientation in the local coordinate system can also be grasped.

[0229] The measurement of the angle of view between the two landmarks 241 from the terminal position X is done twice in one photograph. If the landmark 241 fits in, it can be done by analyzing the photo. A sensor (hereinafter, referred to as a sensor 142) for estimating the orientation of the mobile terminal 100 (the orientation of the terminal in space) Even without a sensor (called a direction sensor), it is possible to estimate the terminal position X and the terminal direction (direction in an external coordinate system). Therefore, it is desirable for the camera 121 to be capable of capturing images of all solid angles. It is nice.

[0230] If the terminal is equipped with a direction sensor, the mobile terminal 100 is rotated to capture multiple images. The terminal position X is calculated based on the direction of each landmark as seen from the internal coordinate system of the mobile terminal 100. The angle between the two landmarks 241 can be measured from the , Only the relative relationship of each landmark 241 direction in the internal coordinate system is used, so the internal coordinate system The relationship between the orientation of the internal coordinate system and the external coordinate system can also be accurately calculated, and this can be used to calibrate the orientation of the internal coordinate system. It is also possible.

[0231] <Modification 4 of the third embodiment> In addition, if the output of the 3-axis gyro sensor 142, which is a direction sensor, is taken into consideration, the landmark 241 is not limited to four points. Three or more points are acceptable as long as all points are not on a straight line. If the position coordinates of the landmark 241 are available, it is possible to calculate the orientation of the plane containing the landmark 241. It is possible. Also, as described later, if the orientation of the internal coordinate system is correct, two or more points can be used. Good too.

[0232] (given three landmarks) The following is a calculation of the terminal position X by the calculation unit 172 when three landmarks 241 are given. In this modification, the mobile terminal 100 uses a three-axis gyro, which is a direction sensor. It uses the output of the sensor 142. This method will be described below.

[0233] First, three landmarks 241 and projection planes 410 and 420 that are not parallel to each other are set. The projection plane 410 and the projection plane 420 only need to be non-parallel, and do not need to be perpendicular to each other. 25 shows an example of three landmarks A, B, and C and a projection plane 410 and a projection plane 420.

[0234] The coordinates of each landmark 241 are projected onto the projection plane 410 and the projection plane 420. The projection points of each point onto the projection plane 410 are designated as points A', B', and C', respectively. The projected points on the surface 420 are points A'', B'', and C'', respectively. The terminal position X is also projected on each surface. Project the points X' and X" respectively.

[0235] From the projection angle (angle) between the landmarks from the terminal position X in the real space, Calculate the angle between each projection point A', B', and C'. The projection direction is the internal coordinate system Since we know the angle θ in the projection plane 410, we can also calculate the angle θ in the projection plane 410.

[0236] The landmarks and the terminal position X projected onto the projection plane 410 are shown in Figs. 26(a) to 26(c). The projection plane 420 is basically the same, so here we will use the projection plane 410 as the Here, we will explain the case where the image is projected onto the

[0237] Since the positional relationships are similar to those in Figs. 23(a), 23(b), and 21(b), respectively, Similarly, the projection plane 41 is calculated from the measured value of the angle of view from the terminal position X to the landmark 241. The terminal position X' within 0 is obtained.

[0238] As shown in FIG. 27, the foot of the perpendicular line drawn from the landmark C' to the straight line A'B' is designated as W'. In addition, if the landmark C' is located farther away than the terminal position X' across the line A'B' (Figure 2 6(a)) is L C’W’ >0,η A’ >0,η B’ >0. Marked point C' is on line A' If the terminal is located closer to the terminal position X' than the terminal position B' (Figure 26(b)), C’W’ < 0,η A’ <0,η B’ <0. When landmark C' is on line A'B' (Figure 26( c)) is L C’W’ = 0. This allows us to unify the formula to the following: cot A’ (L A’B’ -L C’W’ (cotθ A’ +cotθ B’ ))=L B’W ’ cotθ B’ -L A’W’ cotθ A’ ···(77) L A’O’ =L A’W’ -L C’W’ cot A’ (78) L O’X’ =(L A’W’ -L C’W’ cot A’ )(cotθ A’ +cotλ A’ )sinλ A’ (79) In the case of FIG. 26(c), point W' and point O' coincide with each other.

[0239] However, in the projection plane 410, the landmark A'B'C' and the terminal position X' are aligned on a straight line. If they are lined up or on the same circumference, the terminal position X' cannot be obtained. resets or moves the projection plane and performs measurement again.

[0240] This allows the terminal positions X' and X'' projected onto the projection plane 410 and the projection plane 420 to be obtained. Then, the intersection of the straight lines extending from the terminal positions X' and X'' in the projection direction is The terminal position X can be obtained.

[0241] Furthermore, the direction in which the mobile terminal 100 is facing can be estimated from the measurement, so that the direction of the internal coordinate system can be calculated. The orientation can also be corrected. In that case, the terminal position X is calculated using the corrected internal coordinate system in the same procedure. This procedure can be repeated until the terminal position X and the direction of the internal coordinate system reach the required accuracy. It is possible to repeat this process until the desired result is achieved.

[0242] (given two landmarks) Furthermore, the mobile terminal 10 can detect the position of the mobile terminal 10 from the output of the three-axis gyro sensor 142, which is a direction sensor. When the direction based on the internal coordinate system of 0 is obtained, the terminal position can be determined only from the information of the two landmarks 241. X can be calculated.

[0243] For example, as shown in FIG. 28(a), two landmarks 241 (point A and point B) are given. In this case, the terminal position X is the circumference 244 that passes through points A and B, and is rotated around the straight line AB as an axis. The angle AXB(θ AB ) can be calculated from the captured image.

[0244] The direction from the terminal position X to each landmark 241 (V A (o) ,V B (o) ) is known, Terminal position X on the surface (o) can be identified.

[0245] If only two landmarks 241 are given, the landmark orientation (θ AB ) measurement only However, it is not possible to correct the direction of the internal coordinate system of the mobile terminal 100 from the Estimated value of end position X (d) If we obtain the estimate X (d) and the device identified using the above method position (o) The weighted average value of these is set as the terminal position X of the new mobile terminal 100, and the terminal position The direction of the landmark calculated from the position X (V A ) to align with the internal coordinate system.

[0246] (One landmark given) Here, a method for correcting the direction of the internal coordinate system when one landmark 241 is given will be described. In this case, as shown in FIG. 28(b), the terminal position X is on the line L passing through the landmark A. (o ) All I know is that it contains the following. A (o) is the mobile terminal measured in the internal coordinate system. This is the direction of landmark 241 as seen from terminal position X of terminal 100.

[0247] Terminal position X estimated by another method (d) From the line L, the foot of the perpendicular line is X (o) Tosu Furthermore, terminal position X (d) and terminal position X (o) The weighted average of these is taken as the new terminal position X. Then, the direction of the landmark calculated from the terminal position (V A ) to orient the internal coordinate system. Fix it.

[0248] <Modification 5 of the third embodiment> (When landmark measurements are taken at multiple points) In the above embodiment, the owner 910 photographs a plurality of landmarks 241 at one location and However, the present invention is not limited to this. The owner 910 may use different The terminal position X may be calculated using multiple landmarks 241 acquired at the location.

[0249] For example, the owner 910 of the mobile terminal 100 may be in one place and may be able to measure multiple targets required for terminal position estimation. There are cases where it is not possible to photograph the observation point 241. This modified example is designed to deal with such cases. do.

[0250] In this modification, as shown in FIG. 29(a), the owner 910 of the mobile terminal 100 is If it is not possible to capture multiple landmarks 241, the device moves to capture as many landmarks as necessary to estimate the device's position. 241 (in FIG. 29(a), points 241a, 241b, 241c, and 241d) are photographed. At this time, the amount of movement is calculated by an internal function of the mobile terminal 100.

[0251] The calculation unit 172 of this modification calculates the position information of each landmark 241 by calculating the position of the landmark 241 at the same location. The amount of movement (three-dimensional vector quantity) is used to correct the image. The position information of the landmark 241 acquired in step 2 is corrected to position information based on the current position. Then, the terminal position X is calculated using the position information of the corrected landmark 241 .

[0252] Hereinafter, the method of estimating the terminal position X using landmarks 241 acquired at different locations will be referred to as This will be explained using FIG.

[0253] Let the expression of point S in the internal coordinate system of the mobile terminal 100 be Q. S At point S, toward landmark A Suppose you measure the direction of landmark B at point T, and measure the direction of landmark B at point S. Motion Vector D ST is expressed by the following formula: D ST =Q T -Q S (80)

[0254] Movement Vector D ST The amount of movement of the landmark A (coordinate value P A ) to The new landmark point A' (coordinate value P A’ ) as seen from point T The direction is equal to the direction of landmark A from point S. Therefore, the following equation holds: P A’ =P A +D ST (81) V TA’ =V SA (82) Here, the symbol V IJ is the unit vector from point I to point J in the internal coordinate system. .

[0255] In this way, the landmark 241 (point A) measured at a point other than point T is virtually moved to point A'. is set as a new landmark point 241. The amount of movement is set to an amount that cancels the amount of movement between the measurement points. Let us assume that the new landmark 241 (point A') is measured at point T, and the other landmarks measured at point T are Using the identification point 241 (point B), the terminal position X is estimated by the above method.

[0256] In estimating the terminal position X, as described above, the direction and the angle of view between each landmark 241 are calculated. In this modification, the direction and the angle of view between the landmarks 241 are measured by the mobile terminal 10. It can be measured as the direction and angle of view of the landmark in the internal coordinate system of 0.

[0257] In addition, the movement vector between the measurement points is measured by an internal sensor. For example, a 3-axis acceleration sensor The motion vector is calculated as the double integral value of the sensor 143.

[0258] <Application Example 1> As an application of this modified example, the owner 910 of the mobile terminal 100 may move around and Obtain the landmarks necessary for estimating the position, and each time a new landmark is obtained, We will now explain a method for updating the current position using landmarks.

[0259] In this application example, every time the calculation unit 172 estimates the current terminal position X, X is associated with the estimated date and time and managed as estimated location data. The estimated location data is, for example, For example, it is stored in the data storage unit 174.

[0260] An example of the estimated position data 180 is shown in FIG. The data 180 is associated with a point number 181, which is identification information that uniquely identifies each estimated location. The date and time (date / time) when the landmark used to estimate the position was measured 182 and the estimated position The terminal position (point coordinates) 183, the terminal speed at the time of measurement 184, and The coordinates (sign coordinates) 185 of the photographed landmark 241 and the mobile terminal 100 of the landmark 241 The direction from (sign direction) 186 is registered.

[0261] Next, the flow of the terminal position estimation process for calculating the current position of the mobile terminal 100 in the method of this application example will be described. This will be explained. FIG. 31 shows the process flow of the terminal position estimation process of this application example. In this example, the current position is calculated using four landmarks 241. The position coordinates (sign coordinates) of at least four landmarks 241 are obtained, and the terminal position (terminal coordinates) is calculated. It shall be calculated.

[0262] In addition, the terminal position calculated initially by the calculation unit 172 is stored as an estimated position data in association with the calculation time. The positions of the landmarks 241 used in the calculation are registered in the data 180. The information (sign coordinates) and the sign direction in the internal coordinate system of the mobile terminal 100 are also registered.

[0263] The owner 910 carries the mobile terminal 100 and moves a predetermined distance to a new landmark 241. When the owner 910 finds the location, the owner 910 notifies the user of the location via the user I / F 130 of the mobile terminal 100. The system issues an instruction to carry out the set processing and points the camera 121 toward the new landmark point 241.

[0264] When the terminal position estimation unit 170 receives an instruction from the owner 910, the image acquisition unit 171 acquires an image of a new landmark 241 (step S3101).

[0265] When the image acquisition unit 171 acquires an image of a new landmark 241, the calculation unit 172 calculates the image as Analyze and obtain the position information (signal coordinates) of the new sign point 241, and then convert it to the internal coordinate system. Then, the direction (sign direction) of the new landmark 241 is obtained (step S3102).

[0266] The calculation unit 172 further refers to the estimated position data 180 and extracts the date and time of the immediately preceding record. The average speed of the mobile terminal 100 from the date and time to the current time is calculated as the speed. (Step S3103). The velocity is calculated based on the output of the three-axis acceleration sensor 143 for a predetermined time. It is assumed that the time is calculated and recorded at intervals.

[0267] The calculation unit 172 converts the acquired sign coordinates, sign direction, and the calculated speed into new location data. The estimated position data is registered in the estimated position data 180 in association with the new location number (step S3104).

[0268] Next, the calculation unit 172 calculates the terminal position X using the latest four landmarks 241. At this time, in this example, the mobile terminal 100 is located at a different point, except for the new landmark 241. Therefore, first, the motion vector is calculated using the above method. Then, , the sign direction is converted by the movement vector, and the sign assumed to be measured from the current position is The coordinates are obtained (step S3105).

[0269] Then, the estimated position data 180 is updated with the obtained sign direction and sign coordinates (step For example, in the example of FIG. 30(a), the P B1 , P C 1 , P D1 and P E2 The current position is calculated using the above. Therefore, P B1 , P C1 , P D1 of , the current position X according to the amount of movement 2 Convert it into coordinate values ​​based on P B2 , P C2 , P D2 of At this time, the sign direction changes with the terminal position and the sign coordinates, so V 1B , V 1C , V 1D It remains the same.

[0270] The calculation unit 172 calculates the four most recent landmark points (the landmark coordinates P B2 , P C2 , P D2 , P E2 ) to find the current position X 2 Then, the estimated position coordinates are calculated (step S3107). The calculation result is then stored in the estimated position data 180 (step S3108), and the process ends. do.

[0271] Current position 2 After the estimation, the estimated position data 180 is shown in FIG. The record used to calculate the position is rewritten. Also, the information before the rewriting is saved as a history. It may be good to keep it.

[0272] Next, the owner 910 moves, photographs a new landmark 241 (or multiple landmarks) and captures the current location. When an estimation instruction is given, the calculation unit 172 of the mobile terminal 100 estimates the landmark 241 as a point Create and register the No. 3 record. Then, add the last few records to make it a 4-signal point. In the example of the code (Figure 30(b)), the marker coordinates are P C2 , P D2 , P E2 Record of landmarks Then, the converted coordinate position, direction, and new landmark point 24 Using the position and direction information of 1, the terminal position X of the shooting location 3 Calculate.

[0273] <Application Example 2> Furthermore, in this modified example, the owner 910 can check the mark in the captured image every time he / she captures an image. The presence or absence of the point 241 is determined, and if the landmark point 241 is present, it is combined with the landmark points obtained in the past, The terminal position X is calculated, but the number of landmarks used to calculate the terminal position X is not limited to four. Furthermore, even if the same landmark is measured from a different terminal position, the landmark may be different. The process for this case will be described below.

[0274] In this case, for example, the data of the estimated position and the landmark 241 are stored in the data storage device 200 as they are acquired. An example of the estimated location history 190 in this case is shown in FIG. As shown, the estimated location history 190 stores data of almost the same items as the estimated location data 180. Hold.

[0275] That is, each measurement data is associated with data No. 191, which is identification information that uniquely identifies the measurement data. The date and time (date / time) when the landmark used to estimate the location was measured 192 and the estimated location The estimated terminal position (point coordinates) 193, the terminal speed 194 at the time of the estimation, and The coordinates (sign coordinates) 195 of the landmark 241 photographed by the mobile terminal 10 The direction from 0 (sign direction) 196 is registered.

[0276] Next, using this estimated location history 190, the terminal location is estimated using landmarks measured in the past. The flow of the estimation process will be described below. Fig. 33 shows a process flow of the terminal position estimation process in this example. Here, the image acquisition unit 171 starts the process when it acquires a new image. In the case of shooting a video, the process may be started at regular intervals.

[0277] First, the calculation unit 172 analyzes the acquired image and determines whether or not the landmark 241 is detected. (Step S4101). If the moving speed is equal to or lower than a certain value, the object is considered to be stationary. Alternatively, multiple images of the image data taken continuously may be used. For example, the determination is made based on whether or not predetermined position information can be extracted by image analysis.

[0278] If it cannot be detected (step S4101; No), the process ends.

[0279] On the other hand, if one or more landmarks 241 are detected (step S4101; Yes), first , the calculation unit 172 calculates the terminal position X 0 (d) is estimated (step S4 102).

[0280] Here, at the current location, the terminal position X of point No. 1 1 Coordinates (point coordinates) 193 The integrated value of the acceleration of the mobile terminal 100 is added to the velocity 194 of the mobile terminal 100. The integral value of the velocity is calculated by subtracting gravity from the acceleration measurement value by the internal three-axis acceleration sensor 143. Specifically, the coordinates of the terminal position X 0 (d) and device speed S 0 (d) is estimated using the following equation:

number

number

[0281] Next, the calculation unit 172 identifies the number M (M is an integer equal to or greater than 1) of the detected landmarks 241. (Step S4103). Specifically, the calculation unit 172 calculates the terminal position X 0 (d) Taken in The acquired image is analyzed to detect the number of landmarks in the image. The new terminal position estimate X is calculated by combining the information of the landmark 241 and the landmark 241 measured previously. 0 (o) Request.

[0282] Next, the calculation unit 172 calculates the terminal position estimate X0 (o) In calculating the new sign Determine the number of landmarks to be used, N (N is an integer equal to or greater than 1), including the points (step S 4104).

[0283] Specifically, it is determined as follows. (Case 1): Selected in order of newness, including landmark point 241 newly acquired from this image. In a set of four landmarks 241, the distance from the current position to the position of each landmark 241 is obtained. Then, the maximum distance obtained and d max4 and a predetermined value d 4 Compare with d ma x4 is the value d 4 If it is smaller, then the four landmarks 241 are used. (Case 2): If Case 1 is not satisfied, the set of three landmarks 241 selected in order of newness The maximum value of the distance d max3 and a predetermined value d 3 Compare with d max 3 is the value d 3 If it is smaller, then the three landmarks 241 are used. (Case 3): If Case 2 is not satisfied, the distance between the two landmarks selected in the most recent order is The maximum value of d max2 and a predetermined value d 2 Compare with d max2 is the value d 2 Smaller If not, use those two measurement points. (Case 4): If Case 3 is not satisfied either, then only the most recent landmark point 241 is used. The limit value of the distance is set, for example, as follows. However, the present invention is not limited to this. I can't. d 4 <d 3 <d 2 ···(85)

[0284] Once the number of sign points N to be used is determined, the terminal position is calculated using the method according to the number of sign points N described above. The estimated value of X 0 (o) is calculated (step S4105).

[0285] The coordinates of the newly measured landmark 241 are P n At this point, the new internal coordinate system is Let V be the landmark direction vector. n (o) (The landmark 241 measured at the new terminal position is multiplied.) In the case where there are several landmarks, the directions of the landmarks 241 are also included. In the case where the terminal position cannot be calculated, the terminal position is moved and a new measurement is performed. If the conditions are met by using the data from the old measurement point, the measurement point will be used instead. It may be used.

[0286] Next, the weighting coefficient for case n is K n In this case, depending on the case , new terminal position X 0 is calculated using the following formula (step S4106). X 0 =(1-K n )X 0 (d) +K n X 0 (o) ···(86) Here, the weighting coefficients have, for example, the following relationship: 1 ≧ K 4 >K 3 >K 2 >K 1 >0 (87) However, the relationship of the weighting coefficients is not limited to this.

[0287] Next, the direction of the internal coordinate system is updated (step S4107). Here, the external coordinate value is New terminal position coordinate X 0 And the new landmark coordinates P in the external coordinate values n From the terminal position The direction vector V of the new landmark 241 seen from the n Then, calculate the current internal coordinate system The direction vector V of each landmark 241 measured by the standard n (o) and V n From the difference with the internal coordinate system, The measured direction vector is T n The internal coordinate system is rotated so that it is equal to the new target internal coordinate system. Set it as the coordinate system.

[0288] At this time, if there are multiple new landmarks 241, the direction is determined at all of the new landmarks 241. Directional vector V n In some cases, it may not be possible to completely match the calculated values ​​with the measured values. In this case, for example, a new target internal coordinate system is set so that the sum of the directional errors is minimized. The target internal coordinate system may be adopted as the new internal coordinate system, but the weighting coefficient may be used to determine the target internal coordinate system. The rotation angle for forming the coordinate system may be reduced.

[0289] To convert the current internal coordinate system to the target internal coordinate system, for example, rotate the object on three orthogonal axes. For each rotation angle, we use a weighting factor K' n Here, the weight The coefficients have, for example, the following relationship: 1 ≧ K' 4 >K' 3 >K' 2 >K' 1 >0 (88) However, the relationship of the weighting coefficients is not limited to this.

[0290] Finally, the parameters for position estimation by the internal sensor are updated (step S41 08). At this time, point X0 The new estimate of the velocity at is given by: S 0 =S 0 (d) -(X 0 (d) -X 0 ) / Δt (89) Here, Δt is the time lapse from the measurement time of the No. 1 point to the time when the new measurement is taken. It's time.

[0291] Furthermore, the error in the position estimation obtained from the integral of acceleration is primarily due to the fact that the measured acceleration Therefore, the acceleration measurement value is corrected by correcting the gravitational acceleration. The correction is first performed using the following formula before correcting the internal coordinate system. g 0 =g 1 -(S 0 -S 0 (d) ) / Δt (90) where g 0 is the corrected gravitational acceleration. Furthermore, in the new internal coordinate system, g 0 Change the setting.

[0292] Now that the values ​​have been updated, the estimated location history 190 is rewritten. Specifically, Rewrite it in steps.

[0293] First, copy the past data of No.(k) to No.(k+N). The date and time of measurement for data from .(1) to No.(N) as "Date / Time" Time, "Point coordinate" is X 0 , S as "speed" 0 , "label coordinate" as P n , “How to mark "V" as n In addition, we write g, which will be used in the next step in equation (83). 1 and Then, the corrected g0 and the process is terminated.

[0294] After that, the above process is repeated every time an image is captured using the mobile terminal 100.

[0295] As a result, new landmarks are measured while moving, and the terminal's position in the external coordinate system is calculated. It is possible to estimate the direction of the internal coordinate system and perform correction so as to match the direction of the external coordinate system. .

[0296] <Modification 6 of the third embodiment> In this embodiment, the estimation of the terminal position is performed by using a signboard or the like that is previously attached as a sign point 241. For example, the representative points of landmarks, corners of ceilings, etc. In this case, the calculation unit 172 may use these representative points or The position coordinates of the feature points may be obtained from the server 960 and used for position calculation.

[0297] <Seventh Modification of the Third Embodiment> In this embodiment, the position information or the QR code arranged near the landmark 241 is used. The position coordinates of the landmark 241 are acquired from a map, but are not limited to this. The point 241 may be configured to transmit position information (coordinate values).

[0298] In this case, for example, even if the sign 241 emits a signal with good linearity such as light, In addition, the electromagnetic waves emitted from the sign point 241 have good straightness and can be received by a receiver that can measure the direction. If the equipment is available, radio waves may be used. The landmark 241 transmits the beacon signal to the landmark. Continue sending coordinate information of point 241. This allows a compact landmark point 241 to be constructed. Furthermore, the beacon signal may carry an identification signal of the landmark 241. In that case, The signal may be used to obtain location information of the landmark 241 from the server 960 .

[0299] In addition, a beacon signal may be output from a beacon point 241 mounted on the moving object. A moving body carrying the landmark 241 or the landmark 241 itself may notify its own position to other landmarks 24 1. This allows the landmark 241 to automatically calculate its own position. It is configurable and mobile-friendly.

[0300] Furthermore, the mobile terminal 100 itself may be a landmark 241 using a beacon. When terminals are close to each other, it is possible to grasp the precise relative positions of the terminals. This is effective in avoiding collisions when drones are flying close together.

[0301] <Modification> In the above-described embodiments and modifications, the current location is estimated (calculated) within the mobile terminal 100. However, the present invention is not limited to this. 970 and the network 940 to the server 960. In this case, the calculation result may be transmitted to the mobile terminal 100. The message is returned to the original mobile terminal 100 .

[0302] The present invention is not limited to the above-described embodiment and modifications, and various modifications are possible. For example, the above-described embodiment and modifications are included to clearly explain the present invention. The present invention is described in detail for the purpose of convenience, and is not necessarily limited to the embodiment having all of the configurations described. In addition, a part of the configuration of one embodiment or modified example is not intended to be used in conjunction with another embodiment or modified example. In addition, the configuration of one embodiment or modification may be replaced with another embodiment. It is also possible to add the configuration of the embodiment or modification. It is possible to add, delete, or replace part of a configuration with other configurations.

[0303] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be implemented in whole or in part by, for example, The above configurations and functions may be realized by hardware, such as by designing an integrated circuit. The processor interprets and executes the programs that realize the respective functions. It may be realized by software. Each function may be realized by a program, a table, a file, etc. The information is stored in the memory, hard disk, and SSD (Solid State Drive). It can be placed on a recording device such as a digital camera, or on a recording medium such as an IC card, SD card, or DVD. do.

[0304] In addition, control lines and information lines are shown as necessary for explanation, and do not necessarily represent all This does not necessarily show all control and information lines. In reality, almost all components are interconnected. It can be said that this is the case. [Explanation of symbols]

[0305] 100: mobile terminal, 101: CPU, 102: bus, 110: storage device, 111: RO M, 112: RAM, 113: storage, 120: imaging device, 121: camera, 122 : Image processor, 123: Image memory, 130: User I / F, 131: Display 140: sensor device, 141: GPS receiver, 142: 3-axis gyro sensor, 143: 3-axis acceleration sensor, 150: communication device, 151: LAN communication device, 152: telephone network communication device, 153: short-range communication device, 160: expansion I / F, 170: terminal position estimation unit, 171: image acquisition unit acquisition unit, 172: calculation unit, 173: display control unit, 174: data storage unit, 175: QR code analysis section, 176: navigation section, 180: estimated position data, 184: speed, 190: estimated position History, 194: Speed, 200: Sign board, 200a: Sign board, 201: Position and direction information display area, 202: Direction arrow Mark, 211: Sign board, 212: Sign board, 213: Reference point, 221: Sign image, 222: Mark identification image, 230: sign board, 231: sign circle, 231i: sign circle image, 232: azimuth line, 2 32i: Orientation line image, 233: Position and orientation information display area, 234i: Long axis, 240: Sign board ,241: landmark, 242: intersection, 243: location information display area, 244: circumference, 410: Projection surface, 420: Projection surface, 910: Owner, 940: Network, 950: Base station, 960: server, 970: access point

Claims

1. A mobile terminal, A camera and a terminal position estimation unit that processes images acquired by the camera; A three-axis acceleration sensor; A three-axis gyro sensor; The clock section and A distance measuring sensor, The camera captures an image of a signboard attached or installed on a surface of a landmark object whose position information is known, to generate a sign image; the signboard includes position information of the signboard, orientation information of the signboard, and a landmark used for measuring the position of the mobile terminal; The terminal position estimation unit an image acquisition unit for acquiring the sign image; a calculation unit that analyzes the acquired sign image, calculates a plane orientation of a triangle including a side having a length corresponding to the distance from the mobile device to the signboard measured by the distance measuring sensor, a terminal position which is the position of the mobile device, and the landmark, and an interior angle of the triangle, and calculates the terminal position relative to the signboard using the plane orientation and the interior angle; The calculation unit is Based on the moving distance and moving direction detected by the three-axis acceleration sensor, the three-axis gyro sensor, and the clock unit, the moving direction and the amount of movement from the terminal position when the signboard was imaged are detected to estimate the terminal position, and when it is determined that the error in the position estimation has increased, the position of the signboard near the mobile terminal is displayed to prompt the user of the mobile terminal to measure the position, or the mobile terminal automatically measures the position. A mobile terminal characterized by:

2. A mobile terminal, A camera and a terminal position estimation unit that processes images acquired by the camera; A three-axis acceleration sensor; A three-axis gyro sensor; A clock unit, The camera performs a function to calculate a focal length; The camera captures an image of a signboard attached or installed on a surface of a landmark object whose position information is known, to generate a sign image; the signboard includes position information of the signboard, orientation information of the signboard, and a landmark used for measuring the position of the mobile terminal; The terminal position estimation unit an image acquisition unit for acquiring the sign image; a calculation unit that analyzes the acquired sign image, calculates a surface orientation of a triangle including a side having a length corresponding to a distance from the mobile device to the signboard calculated based on a focal length when the camera captured the image of the signboard, a terminal position which is the position of the mobile device, and the landmark, and an interior angle of the triangle, and calculates the terminal position relative to the signboard using the surface orientation and the interior angle; The calculation unit is Based on the moving distance and moving direction detected by the three-axis acceleration sensor, the three-axis gyro sensor, and the clock unit, the moving direction and the amount of movement from the terminal position when the signboard was imaged are detected to estimate the terminal position, and when it is determined that the error in the position estimation has increased, the position of the signboard near the mobile terminal is displayed to prompt the user of the mobile terminal to measure the position, or the mobile terminal automatically measures the position. A mobile terminal characterized by:

3. 3. The mobile terminal according to claim 1, Further comprising a display control unit that displays the calculated terminal position on a display of the mobile terminal. A mobile terminal characterized by:

4. 3. The mobile terminal according to claim 1, The terminal position estimation unit further includes a navigation unit that performs navigation, The calculation unit outputs the calculated terminal position to the navigation unit. A mobile terminal characterized by:

5. 3. The mobile terminal according to claim 1, The landmark is a representative point of the landmark. A mobile terminal characterized by:

6. 3. The mobile terminal according to claim 1, The device further includes a display that functions as both a reception unit that receives a target point setting from a user and a display unit, Based on the calculated terminal position and the orientation of the mobile terminal, an image showing the three-dimensional position of the target point is superimposed on the actual image acquired by the camera and displayed. A mobile terminal characterized by:

7. A terminal position estimation system comprising: a mobile terminal equipped with a camera; and a server that processes images acquired by the camera, the system estimating a terminal position, which is the position of the mobile terminal, by analyzing the images, the mobile terminal includes a three-axis acceleration sensor, a three-axis gyro sensor, a clock unit, and a distance measurement sensor; The camera captures an image of a signboard attached or installed on a surface of a landmark object whose position information is known, to generate a sign image; the signboard includes position information of the signboard, orientation information of the signboard, and a landmark used for measuring the terminal position; The server, an image acquisition unit that acquires the sign image acquired by the camera; a calculation unit that analyzes the sign image, calculates a plane orientation of a triangle including a side having a length corresponding to the distance from the mobile device to the signboard measured by the distance measuring sensor, the terminal position, and the landmark, and an interior angle of the triangle, and calculates the terminal position relative to the signboard using the plane orientation and the interior angle; The calculation unit is Based on the moving distance and moving direction detected by the three-axis acceleration sensor, the three-axis gyro sensor, and the clock unit, the moving direction and the amount of movement from the terminal position when the signboard was imaged are detected to estimate the current terminal position, and if it is determined that the error in the position estimation has increased, the position of the signboard near the mobile terminal is displayed to prompt the user of the mobile terminal to measure the position, or the server automatically measures the position. A terminal position estimation system comprising:

8. A terminal position estimation system comprising: a mobile terminal equipped with a camera; and a server that processes images acquired by the camera, the system estimating a terminal position, which is the position of the mobile terminal, by analyzing the images, the mobile terminal includes a three-axis acceleration sensor, a three-axis gyro sensor, a clock unit, and a focal length calculation function unit; The camera captures an image of a signboard attached or installed on a surface of a landmark object whose position information is known, to generate a sign image; the signboard includes position information of the signboard, orientation information of the signboard, and a landmark used for measuring the terminal position; The server, an image acquisition unit that acquires the sign image acquired by the camera; a calculation unit that analyzes the sign image, calculates a surface orientation of a triangle including a side having a length corresponding to a distance from the mobile device to the signboard, the side being calculated based on a focal length when the camera captured the image of the signboard, the terminal position, and the landmark, and an interior angle of the triangle, and calculates the terminal position relative to the signboard using the surface orientation and the interior angle; The calculation unit is Based on the moving distance and moving direction detected by the three-axis acceleration sensor, the three-axis gyro sensor, and the clock unit, the moving direction and the amount of movement from the terminal position when the signboard was imaged are detected to estimate the current terminal position, and if it is determined that the error in the position estimation has increased, the position of the signboard near the mobile terminal is displayed to prompt the user of the mobile terminal to measure the position, or the server automatically measures the position. A terminal position estimation system comprising:

9. A method for estimating a terminal position, which is the position of a mobile terminal, by using a server that processes an image acquired by a camera mounted on the mobile terminal, comprising: A step of acquiring a sign image generated by capturing an image of a signboard, which is attached or installed on a surface of a landmark object whose position information is known, and which includes position information and orientation information of the signboard and a landmark point used for measuring the position of the mobile device, with a camera mounted on the mobile device; analyzing the acquired sign image, calculating a side length corresponding to the distance from the mobile terminal to the signboard measured by a distance measuring sensor mounted on the mobile terminal or the distance from the mobile terminal to the signboard calculated based on the focal length when the camera captured the image of the signboard, a plane orientation of a triangle including a terminal position, which is the position of the mobile terminal, and the landmark, and an interior angle of the triangle, and calculating the terminal position relative to the signboard using the plane orientation and the interior angle; A step of detecting a direction and amount of movement from the terminal position when the image of the signboard was captured based on a distance and direction of movement detected by a three-axis acceleration sensor, a three-axis gyro sensor, and a clock unit mounted on the mobile terminal, and estimating a current terminal position; a step of indicating the location of the signboard near the mobile terminal to prompt the user of the mobile terminal to measure the location when it is determined that the error in the location estimation has increased; A terminal position estimation method comprising:

10. A terminal position estimation method for estimating a terminal position, which is the position of a mobile terminal, by using a server that processes an image acquired by a camera mounted on the mobile terminal, comprising: A signboard is attached or installed on a surface of a landmark object whose position information is known, and the signboard includes position information and orientation information of the signboard and a landmark point used for measuring the position of the mobile device. A signboard image is acquired by capturing the image of the signboard using a camera mounted on the mobile device. analyzing the acquired sign image, calculating a side length corresponding to the distance from the mobile terminal to the signboard measured by a distance measuring sensor mounted on the mobile terminal or the distance from the mobile terminal to the signboard calculated based on the focal length when the camera captured the image of the signboard, a plane orientation of a triangle including a terminal position, which is the position of the mobile terminal, and the landmark, and an interior angle of the triangle, and calculating the terminal position relative to the signboard using the plane orientation and the interior angle; A step of detecting a direction and amount of movement from the terminal position when the image of the signboard was captured based on a distance and direction of movement detected by a three-axis acceleration sensor, a three-axis gyro sensor, and a clock unit mounted on the mobile terminal, and estimating a current terminal position; when it is determined that the error in the location estimation has become large, the server automatically performs location measurement; A terminal position estimation method comprising:

11. 3. The mobile terminal according to claim 1, The calculation unit detects a moving distance and a moving direction by the three-axis acceleration sensor, the three-axis gyro sensor, and the clock unit, and displays the corrected values ​​on the map information. A mobile terminal characterized by:

12. 9. A terminal position estimation system according to claim 7 or 8, The mobile terminal detects a moving distance and a moving direction by the three-axis acceleration sensor, the three-axis gyro sensor, and the clock unit, and displays the corrected values ​​on the map information. A terminal position estimation system comprising:

13. 11. A terminal position estimation method according to claim 9, further comprising: The mobile terminal further includes a step of detecting a moving distance and a moving direction by the three-axis acceleration sensor, the three-axis gyro sensor, and the clock unit, and indicating the corrected values ​​in map information. A terminal position estimation method comprising:

Images