Route guidance device, route guidance system, and program

Abstract

To provide a route guide device capable of providing route guidance in various spaces including indoors, and a route guidance system and a program.SOLUTION: A route guide device is configured to perform the steps of: acquiring a piece of map information in which a position where a key image is displayed is specified by the map coordinate system; detecting a key image from image data which is picked up from the vicinity of the route guide device; and acquiring relative position information between the key image in a given virtual space coordinate system and the route guide device on the basis of the detected key image. The route guide device is further configured to acquire position information in the map of the map coordinate system of the route guide device using the acquired map information and the acquired relative position information, and provide the acquired position information in the map for predetermined processing to search and guide the route.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a route guidance device, a route guidance system, and a program.

Background Art

[0002] In recent years, route guidance systems using GPS (Global Positioning System) have become widespread, and in places where GPS signals can be received, such as outdoors, route guidance systems using application programs operating on smartphones and the like are widely used.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In this conventional route guidance system, it was impossible to perform route guidance in places where GPS signals cannot be received, such as indoors. Therefore, a technique for guiding a route using a beacon signal, as disclosed in Patent Document 1, for example, has been considered.

[0005] However, in the route guidance technique using a beacon signal, information on the relative position of the user trying to receive the guidance, such as the orientation with respect to the beacon signal transmitter, cannot be obtained, so there are limitations in the guidance method.

[0006] In view of the above circumstances, an object of the present invention is to provide a route guidance device, a route guidance system, and a program capable of performing route guidance in various spaces including indoors.

Means for Solving the Problems

[0007] One aspect of the present invention for solving the problems of the above conventional examples is a route guidance device for guiding a route combining edges, wherein a predetermined key image is posted at a plurality of locations near the edges, and map information defining the edges in a predetermined map coordinate system, at least the positions where the key images are posted are map information acquisition means for acquiring map information defined by the map coordinate system, imaging means for imaging the periphery of the route guidance device and outputting the captured image data, key detection means for detecting the key image from the image data, and based on the detected key image, relative position acquisition means for acquiring relative position information between the key image in a predetermined virtual space coordinate system and the route guidance device, and map internal position information acquisition means for acquiring the internal position information of the map of the route guidance device in the map coordinate system using the acquired map information and the acquired relative position information, and the acquired internal position information of the map is subjected to a predetermined process of searching for and guiding a route along the edge.

Effects of the Invention

[0008] According to the present invention, it is possible to perform route guidance in various spaces including indoors.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0010] Embodiments of the present invention will be described with reference to the drawings. A route guidance system 1 including a route guidance device 10 according to an embodiment of the present invention is used in a facility or the like where a predetermined key image is posted near an edge, with a route such as a passage through which a user can pass as the edge.

[0011] Here, the key image is an image represented on a rectangular plane and may include, for example, a bar code or a computer-readable code image such as a two-dimensional bar code. However, the key image is not limited to these examples, and the key image may be, for example, an advertisement or a signboard of a store itself. In the example of this embodiment, the key image is supported by a support member such as a frame for posting the key image on a wall surface or the like near the edge. Hereinafter, this key image and the support member are referred to as a key image posting body.

[0012] As illustrated in FIG. 1, the route guidance system 1 of this embodiment includes a route guidance device 10 carried by a user and a map information server 20 that is connected to be accessible to the route guidance device 10 via a network.

[0013] In addition, in the example of this embodiment, the administrator of the route guidance system 1 generates map information in advance. As illustrated in FIG. 2(a), this map information includes a map image (which may be a rasterized image or a vector image) of the facility or area where route guidance is provided, and edges and nodes (endpoints of the edges) defined along the routes that users can pass through in the facilities or areas shown in the map image. In the example of this embodiment, the edges and nodes are defined by coordinates within a predetermined map coordinate system.

[0014] In addition, it is assumed that in this map information, the position and orientation where the key image is posted are defined as coordinates or vectors in this map coordinate system. Here, for the map coordinate system, for example, with a predetermined position on the ground surface as the origin, an X′Z′ orthogonal coordinate system is set on a plane parallel to the ground surface (assuming the ground surface is approximately planar), and a right-handed X′Y′Z′ orthogonal coordinate with the direction perpendicular to the ground surface (with the vertically upward direction being the positive direction) as the Y′ axis may be used. At this time, the positive direction of the X′ axis or the positive direction of the Z′ axis may be set as the north direction.

[0015] Specifically, in the example of FIG. 2(a), a plurality of edges Ea, Eb,... and nodes Ma, Mb,... which are the endpoints of the edge E are set on the passage P where users can pass through. Also, in the example of FIG. 2, the posting position Q of the key image in the map coordinate system near the edge E and the information S for identifying the key image posted at the posting position are associated and recorded.

[0016] Furthermore, as shown in FIG. 2(b), the map information includes information representing the posting position Q of the key image in the map coordinate system, the direction Qd of the key image in the map coordinate system (which may be the normal direction to the plane of the key image), and information representing the actual size Qs of the key image, etc., associated with the information S for identifying the key image.

[0017] In addition to these, positions that users can set as destinations, such as the positions of stores within the facility (represented in the map coordinate system), etc., may be preset in this map information.

[0018] Here, the information for specifying the key image may be the decoded data obtained by decoding the code image when the code image is included in the key image. Further, when the key image does not include the code image, any information necessary for searching for the key image from the captured image data, such as the key image itself or the feature amount obtained based on the key image (since various well-known methods can be adopted for searching for the image portion serving as a predetermined key from the image, detailed description thereof is omitted here), may be used.

[0019] Note that, since this shows a case where there are a plurality of floors connected through an escalator, an elevator, stairs, etc., map information corresponding to each floor and information for specifying node M corresponding to each other on each floor are set. As an example, in FIG. 2, map information A of the first floor and map information B of the second floor are shown, and a state where node Mx (the midpoint of the escalator) is set in both at corresponding positions is shown.

[0020] In the present embodiment, the map information is stored in the map information server 20. Specifically, the map information server 20 stores the map information in association with information for specifying facilities, buildings, etc. corresponding to the map information. This map information server 20 receives a request for map information via a network together with information for specifying facilities, buildings, etc. The map information server 20 reads out the map information corresponding to the facility or building specified by the information received together with the request in response to the request for the map information, and sends the read map information to the requesting smartphone or the like via the network.

[0021] Furthermore, the route guidance system 1 of the present embodiment includes a route guidance device 10 carried by a user. This route guidance device 10 may be, for example, a smartphone or the like, and specifically, as illustrated in FIG. 1, includes a control unit 11, a storage unit 12, an operation unit 13, a display unit 14, an imaging unit 15, and a communication unit 16.

[0022] The control unit 11 is a program control device such as a CPU, and operates according to a program stored in the storage unit 12. In the present embodiment, this control unit 11 performs processes for fulfilling the original functions of a smartphone or the like that operates as the route guidance device 10, and also accesses the map information server 20 to acquire map information and perform the next route guidance process.

[0023] That is, the control unit 11 of the route guidance device 10 according to the example of the present embodiment receives image data obtained by imaging the periphery of the route guidance device 10 itself from the imaging unit 15, and executes a process for detecting a key image from the image data. Here, when a key image is detected, the control unit 11 acquires information representing the relative position information between the key image in a predetermined virtual space coordinate system and the route guidance device 10 based on the detected key image.

[0024] Further, the control unit 11 uses the acquired map information and relative position information to acquire the in-map position information of the route guidance device 10 in the map coordinate system, and subjects the acquired in-map position information to a predetermined process of searching for and guiding a route along the edge. The specific content of the process of the control unit 11 for route guidance will be described later.

[0025] The storage unit 12 is a memory device or the like, and holds programs and the like executed by the control unit 11. This program may be stored in a computer-readable and non-transitory recording medium and provided, and may be copied to this storage unit 12, or may be downloaded via a network. Further, in the present embodiment, this storage unit 12 also operates as a work memory of the control unit 11.

[0026] The operation unit 13 is a touch panel or the like, receives an instruction operation from a user, and outputs information representing the content of the instruction operation to the control unit 11. The display unit 14 is a display or the like laminated below the touch panel, and outputs information for display according to an instruction input from the control unit 11.

[0027] The imaging unit 15 is a camera or the like, sequentially acquires images of the periphery of the route guidance device 10, and sequentially outputs the data of the acquired images to the control unit 11.

[0028] The communication unit 16 includes a wireless network interface. This communication unit 16 outputs information received wirelessly via the network to the control unit 11. Also, the communication unit 16 transmits information via the network according to an instruction input from the control unit 11. Note that this communication unit 16 may be configured to perform communication via a mobile phone network if the route guidance device 10 is, for example, a smartphone.

[0029] Next, the content of the specific route guidance process by the control unit 11 of the route guidance device 10 will be described. As illustrated in FIG. 3, the control unit 11 that performs this process is functionally configured to include a map information acquisition unit 21, a key detection unit 22, a relative position acquisition unit 23, an in-map position information acquisition unit 24, a route guidance processing unit 25, and an information output unit 26.

[0030] In the following description, the control unit 11 sets a virtual space coordinate system with XZ orthogonal coordinates on a plane parallel to the actual floor surface and the direction perpendicular to this plane (with the upward direction as the positive direction) as the Y axis, and performs processing using this virtual space coordinate system and the map coordinate system which is the coordinate system of the map information. Also, here, for example, when the positive direction of the X axis or Z axis of the virtual space coordinate system is set as the right-handed XYZ coordinate system with the direction in which the route guidance device 10 is facing (the central direction of the imaging angle of the imaging unit 15) at the time when the user activates the route guidance device 10 as the positive direction of the X axis, the virtual space coordinate system can be used. However, the method of setting the virtual space coordinate system is not limited to this. For example, when the north direction can be detected by a geomagnetic sensor or the like built in the route guidance device 10, the north direction may be set as the positive direction of the X axis.

[0031] The map information acquisition unit 21 accesses the map information server 20 to acquire map information. In an example of this embodiment, the user inputs the name of the facility where he / she is located. The map information acquisition unit 21 requests the map information server 20 for the map information corresponding to the facility with the input facility name. When the map information acquisition unit 21 receives the map information from the map information server 20, it stores the map information in the storage unit 12.

[0032] During the route guidance process, the key detection unit 22 controls the imaging unit 15 to sequentially image the periphery of the route guidance device 10 and outputs the image data obtained by the imaging. The key detection unit 22 receives the image data sequentially imaged and output by the imaging unit 15, and detects a key image from the image data. The key detection unit 22 outputs information representing the position (which may be a position in the XZ plane) and the orientation (the normal direction with respect to the plane of the key image) of the detected key image in the virtual space coordinate system.

[0033] When the key image includes a computer-readable code image, the key detection unit 22 detects the computer-readable code image that becomes the key image from the captured image data. For the process of detecting the code image from such image data, a widely known method can be adopted, so the detailed description here is omitted.

[0034] The key detection unit 22 decodes the code image included in the detected key image to obtain decoded data. Then, using the decoded data as a key, it searches for and acquires information representing the position of the detected key image on the map coordinates of the map information acquired by the map information acquisition unit 21.

[0035] Then, the key detection unit 22 outputs the position of the acquired key image on the map coordinates, the position (which may be a position in the XZ plane) and the orientation (the normal direction with respect to the plane of the key image) in the virtual space coordinate system.

[0036] When the key image does not include the code image, the key detection unit 22 attempts to detect the key image from the captured image data using information for specifying the key image included in the map information acquired by the map information acquisition unit 21. Specifically, the key detection unit 22 executes a process (image search process) of detecting a portion that matches the key image included in the acquired map information.

[0037] When any one of the key images included in the map information is detected, the key detection unit 22 outputs information representing the position on the map coordinates of the detected key image (obtained from the map information), the position in the virtual space coordinate system (which may be the position in the XZ plane), and its orientation (the normal direction with respect to the plane of the key image).

[0038] Based on the information on the position and orientation of the key image detected by the key detection unit 22, the relative position acquisition unit 23 acquires relative position information between the key image in the virtual space coordinate system and the route guidance device 10. Specifically, first, based on the information on the area in the image data acquired by the imaging unit 15 where the key image is captured, and the information on the position ((x2, y2, z2)) and orientation of the key image, the relative position acquisition unit 23 estimates the position in the virtual space coordinate system of the imaging unit 15, that is, the position (x1, y1, z1) in the virtual space coordinate system of the route guidance device 10. As a method for this estimation, a widely known method using virtual reality technology (so-called AR technology) can be adopted. An example of this method corresponds to the method for obtaining the camera coordinate system in, for example, Hirokazu Kato, et.al., "An Augmented Reality System Based on Marker Tracking and Its Calibration", Journal of the Virtual Reality Society of Japan, Vol4, No4, pp.607-616, (1999). Note that for the coordinates y1 and y2 in the height direction, both may be set in advance to a value h representing the line-of-sight height.

[0039] The in-map position information acquisition unit 24 generates information representing the position of the route guidance device 10 in the map coordinate system based on the relative position between the captured key image and the route guidance device 10 obtained by the relative position acquisition unit 23. As an example, the in-map position information acquisition unit 24 obtains information representing the relative position between the key image in the virtual space coordinate system and the route guidance device 10: x2 - x1 y2 - y1 z2 - z1 to obtain.

[0040] Here, as illustrated in FIG. 4(a), let the position of the key image in the XZ plane of the virtual space coordinate system be B(x2, z2) and the position of the route guidance device 10 be A(x1, z1). The distance d(AB) between A and B is

Equation

Equation

[0041] In the following description, the sign of the angle changes depending on whether it is measured clockwise or counterclockwise when viewing the XZ plane from the positive Y-axis direction. That is, angle CBA = -angle ABC.

[0042] Furthermore, taking a point D on the line segment passing through point B and parallel to the direction of the key image (the normal direction of the plane of the key image) (assuming the point is closer to point A than point B), [Number] holds. Note that angle CBD is known from the settings.

[0043] Since the information regarding this angle also holds in the real space, ultimately, as illustrated in FIG. 4(b), even in the X′Z′ plane in the map coordinate system X′Y′Z′, with the position of the key image being B′(x′2, z′2) and the position of the route guidance device 10 being A′(x′1, z′1), and further, when a point (a point closer to point A′ than point B′) on the line segment parallel to the direction of the key image (the normal direction of the surface of the key image) passing through point B′ is defined as D′, [Number] holds.

[0044] Here, s is the scale ratio of the map coordinate system with respect to the virtual space coordinate system, and point C′ is, in the map coordinate system, a point on the line segment parallel to the Z′ axis passing through point B′ (a point in the positive direction of the Z′ axis). Note that the scale ratio s can be obtained by the ratio between the size of the key image in the virtual space coordinate system (for example, its width, which can be obtained as the difference in coordinate values in the virtual space coordinate system at both ends in the width direction) and the size of the key image in the map coordinate system (the corresponding direction, for example, the difference in coordinate values in the map coordinate system at both ends in the width direction in the above example).

[0045] Also, from this equation (6), [Number] is obtained, and the coordinates (x′1, z′1) of the position A′ of the route guidance device 10 in map coordinates can be obtained as x′1 = x′2 + Δx′, z′1 = z′2 + Δz′ can be obtained.

[0046] The in-map position information acquisition unit 24 obtains the values of formulas (1) to (3) based on the information on the relative position between the captured key image and the route guidance device 10, and obtains Δx′ and Δz′ by formulas (4) to (8) using the separately obtained scale ratio s and these obtained values. Then, the in-map position information acquisition unit 24 uses the position coordinates (x′2, z′2) of the captured key image in map coordinates, x′1 = x′2 + Δx′, z′1 = z′2 + Δz′ and obtains the coordinates (x′1, z′1) of the position A′ of the route guidance device 10 in map coordinates.

[0047] The route guidance processing unit 25 accepts the setting of the destination from the user and performs processing for guiding the route. Specifically, this route guidance processing unit 25 uses the position of the route guidance device 10 in map coordinates (that is, the position of the user carrying it) acquired by the in-map position information acquisition unit 24 to obtain the point CU on the edge closest to the position (the point representing the current position) on the map information.

[0048] Then, the route guidance processing unit 25 obtains the route from the current position to the destination using the point DE on the edge of the set destination and the point CU on the edge representing the current position. Since the method of obtaining the route on the edge from the current position to the destination based on these two points on the edge is widely known, detailed description here is omitted.

[0049] The route guidance processing unit 25 instructs the information output unit 26 to output the information on the route obtained here.

[0050] The information output unit 26 is realized by using library software for general virtual reality (AR) processing. Examples of such library software include ARKit (Apple Inc., USA) and AR Core (Google Inc., USA).

[0051] This information output unit 26 uses such library software to display the image data obtained by the imaging unit 15 on the display unit 14, and at the same time, superimposes a guidance image for guiding the route on this image data and displays it. Specifically, as illustrated in FIG. 5, this guidance image is a virtual reality image (three-dimensional image) including a straight line E (virtual straight line) representing an edge defined by map information and an arrow P (virtual arrow-shaped figure) representing the direction in which the user should proceed.

[0052] Such a guidance image is obtained by converting an edge defined by a plurality of points on the map coordinates in the map information into points in a virtual space coordinate system that can be superimposed on the image data obtained by the imaging unit 15, and drawing a virtual straight line connecting the converted points as an image seen from the position of the route guidance device 10 in the virtual space coordinate system.

[0053] Specifically, the conversion from the map coordinate system to the virtual space coordinate system is performed in the same manner as the example of the conversion from the virtual space coordinate system to the map coordinate system exemplified above. Specifically, when there is a route guidance device 10 at point A′(x′1, z′1) on the map coordinate system illustrated in FIG. 6(a), the positions of the endpoints (nodes) M′1 and M′2 of the edge E12 in the virtual space coordinate system are obtained as follows.

[0054] First, the information output unit 26 performs the following processing for each of the points M′1 and M′2 in the map coordinates. That is, the information output unit 26 sets a point C′ in the positive direction of the Z′ coordinate passing through the point M′ (either M′1 or M′2, M′1 is taken as an example in FIG. 6), and also sets a point D′ on the line segment orthogonal to the edge E12 passing through this point M′ and closer to the point A′ than the point M′. Hereinafter, the coordinates of the point M′ are represented as (x′3, z′3).

[0055] And the information output unit 26 calculates the distance d(A′M′) between the point A′ and the point M′ as

Equation

Equation

[0056] Since the angle is preserved before and after the conversion, the information output unit 26 determines the relationship between the angles of each point and the distance,

Number

[0057] And the information output unit 26

Number

[0058] The information output unit 26 obtains the coordinate values in the virtual space coordinate system for each of the point M'1 and point M'2 in the map coordinates by the conversion using the equations from (9) to (16). Then, a virtual straight line connecting these coordinate values is drawn and rendered as an image seen from the position of the route guidance device 10 in the virtual space coordinate system. This process is the same as the rendering process in virtual reality (AR).

[0059] Further, the information output unit 26 may draw a virtual arrow in the direction of point M′1 or point M′2 (the direction of the node that the user should head towards) by superimposing it on a virtual straight line or the like drawn here. This process can be performed, for example, by drawing a conical figure.

[0060] [Operation] This embodiment basically has the above configuration and operates as follows. When a user carrying the route guidance device 10 visits a facility where a key image is posted at various locations, starts the route guidance device 10, and captures one of the key images, the route guidance device 10 starts the process illustrated in FIG. 6 below. In the following example, it is assumed that the key image includes a code image such as a two-dimensional barcode, and information for specifying map information (map information specifying information) and information for specifying the key image are encoded and represented in the code image. Here, the map information includes information on edges representing routes that the user can pass through and information such as the position of the key image, and these information are represented in a map coordinate system.

[0061] First, the route guidance device 10 recognizes the key image from the captured image data and extracts the code image included in the key image. The route guidance device 10 decrypts the extracted code image to obtain decrypted data. As described above, this decrypted data includes map information specifying information and information for specifying the key image.

[0062] The route guidance device 10 acquires the map information specified by the map information specifying information included in the decrypted data from the map information server 20 as illustrated in FIG. 7 (S1). Further, the route guidance device 10 detects the position and orientation information of the key image included in the captured image data (S2), arranges XZ orthogonal coordinates on a plane parallel to the actual floor surface, and acquires the relative position information between the key image in the virtual space coordinate system with the direction perpendicular to this plane (the upward direction being the positive direction) as the Y axis and the route guidance device 10 (S3).

[0063] Next, the route guidance device 10 generates information representing the position of the route guidance device 10 in the map coordinate system based on the captured key image and the relative position between the route guidance device 10 and the key image (S4). The information on the position of the route guidance device 10 in the map coordinate system can be obtained by the method described above. That is, the route guidance device 10 obtains information on the position and orientation of the key image in the map coordinates specified by the information specifying the key image in the decoded data from the map information acquired in step S1. Further, the route guidance device 10 obtains information on the position of the route guidance device 10 in the map coordinate system based on the relative position information between the key image in the virtual space coordinate system acquired in step S3 and the route guidance device 10, and the information on the position and orientation of the key image in the map coordinates.

[0064] This process may also obtain information representing the position of the route guidance device 10 in the marker coordinate system by converting the marker coordinate system with the key image as a marker and a given virtual space coordinate system, and converting the marker coordinate system and the camera coordinate system which is the coordinate system seen from the route guidance device 10, obtain the conversion formula between the position and orientation in the map coordinate system of the corresponding key image and the marker coordinate system, and obtain the information representing the position of the route guidance device 10 in the map coordinate system.

[0065] The route guidance device 10 also receives information on the coordinates in the map coordinate system representing the destination location from the user (S5). Then, the route guidance device 10 determines the point DE on the edge closest to the input destination location and sets the destination (S6). Note that the input of the destination in step S5 may be performed, for example, by referring to a destination database in which store names and information on their positions in the map coordinate system at the entrances are associated in advance, displaying a list of store names included in the destination database, and receiving the selection of a store from the user from among the list. In this example, the route guidance device 10 performs the process of step S6 using the information on the position in the map coordinate system associated with the selected store name as the coordinates of the destination.

[0066] The route guidance device 10 determines the point CU on the edge closest to the coordinates in the map coordinate system represented by the information generated in step S4 and sets the current position (S7), and generates a route from the point CU representing the current position to the point DE representing the destination through the edge and executes route guidance processing (S8). The route generated here can be represented, for example, as a permutation of the edges and nodes to be passed through until reaching the destination, and the processing for generating it can use the processing in a widely known navigation system.

[0067] Also, an example of the route guidance processing in step S8 is as illustrated in Fig. 8(a). Note that the route guidance device 10 repeatedly performs the process of capturing an image by the imaging unit 15 during this route guidance processing.

[0068] As route guidance processing, the route guidance device 10 synthesizes and displays, on the image data obtained by the imaging unit 15 capturing an image, an image in which the edge including the point CU is drawn as a line segment in the virtual space coordinate system (S11). Also, in this step S11, the route guidance device 10 may also draw, together with the image of the straight line representing the edge, an image of an arrow indicating the direction of the next node to be passed through.

[0069] Specifically, in this example, the route guidance device 10 obtains the coordinates in the virtual space coordinate system of the endpoints of the edge including the point CU. In this process, the route guidance device 10 converts the position of the endpoint of the edge (represented in the map coordinate system in the map information) into the information of the virtual space coordinate system, and draws the straight line connecting the converted points as an image seen from the position of the route guidance device 10 in the virtual space coordinate system. Here, the process of superimposing and drawing a straight line or the like connecting each point of the virtual space coordinate system set based on the captured image data on the image data can be realized by using library software for AR processing.

[0070] When the user moves while referring to the image displayed on the route guidance device 10, the route guidance device 10 uses library software for AR processing to obtain the position of the route guidance device 10 in the virtual space coordinate system after moving with the user by motion tracking processing (S12). Then, the route guidance device 10 returns to step S11, and superimposes on the image data last captured by the imaging unit 15, and newly draws a guidance image including a virtual straight line representing an edge, a virtual arrow representing the direction of the next node to pass through, etc. as a guidance image from the position updated in step S12. The route guidance device 10 repeats the processing of steps S11 and S12 until the user instructs to stop the processing or until the distance between the position acquired in step S12 and the destination falls below a predetermined threshold value.

[0071] [Position correction during movement] Also, when the user moves during the route guidance process by the route guidance device 10, not only motion tracking processing is performed, but also if a new key image can be detected from the image data captured by the imaging unit 15, based on information such as the position and orientation of the newly detected key image, the position information of the route guidance device 10 in the map coordinate system may be corrected.

[0072] As another example of the route guidance process in step S9, the route guidance device 10 in this example sequentially receives the image data obtained by sequentially imaging by the imaging unit 15, and synthesizes and displays an image in which an edge including the point CU is drawn as a line segment in the virtual space coordinate system for the received image data (S21). This process is the same as the process of step S11 shown in FIG. 8(a). Also in this step S21, the route guidance device 10 may also draw an image of an arrow indicating the direction of the next node to pass through together with the image of the straight line representing the edge.

[0073] Also, the route guidance device 10 attempts to detect a key image from the image data captured by the imaging unit 15 (S22). Here, when the route guidance device 10 detects information on the position and orientation of the key image included in the captured image data (S22: Yes), it uses the detected information on the position and orientation of the key image to acquire relative position information between the key image in the virtual space coordinate system and the route guidance device 10 (S23).

[0074] Then, based on the relative position between the captured key image and the route guidance device 10, the route guidance device 10 generates information representing its position in the map coordinate system of the route guidance device 10 (S24). Since the information on the position of the route guidance device 10 in this map coordinate system is the same as the method described above, repeated explanation is omitted. The route guidance device 10 will thereby acquire its position in the virtual space coordinate system after moving with the user. The position information obtained in this step S24 is position information corrected by the newly detected key image and is expected to be relatively accurate compared to the position obtained by the motion tracking process.

[0075] On the other hand, if the route guidance device 10 fails to detect a key image from the captured image data in step S22 (S22: No), it uses library software for AR processing to acquire, by motion tracking processing, its position in the virtual space coordinate system after moving with the user (S25). Then, the route guidance device 10 returns to step S21 and newly draws a guidance image including a virtual straight line representing an edge, a virtual arrow representing the direction of the next node to pass through, etc. as a guidance image from the position updated in step S24 or step S25, superimposed on the image data last captured by the imaging unit 15.

[0076] According to this example of the present embodiment, the error in the position information accumulated by the motion tracking process accompanying the movement of the user can be corrected based on the relative position information with the newly detected key image.

[0077] [Presentation of Guidance Information] In addition, when the route guidance device 10 of the present embodiment displays information including a guidance image, it may determine whether a predetermined stop condition is satisfied, and at least stop displaying the guidance image while the stop condition is satisfied.

[0078] Here, the stop condition may be a condition based on the current position of the route guidance device 10 and a region (hereinafter referred to as a stop condition region) predetermined in the map information.

[0079] In this example, the administrator of the route guidance system 1 sets a stop condition region in the map information in advance. This stop condition region can be represented as a set of three or more points (stop condition region specified points) surrounding the stop condition region. Here, the coordinates of the stop condition region specified points are represented in the map coordinate system.

[0080] When the route guidance device 10 displays information including a guidance image, for example, when performing the processing of step S11 or step S21 in FIGS. 8(a) and 8(b), the route guidance device 10 checks whether a point CU in the map coordinate system representing the position of the route guidance device 10 at that time is inside the above stop condition region. When the route guidance device 10 determines that the point CU in the map coordinate system representing the position of the route guidance device 10 is inside the above stop condition region, at least the display of the guidance image including a straight line representing an edge and an arrow representing the direction of the next node to pass through is stopped.

[0081] At this time, the route guidance device 10 may also stop displaying the image data last captured by the imaging unit 15 in addition to the guidance image. Further, when stopping the display of the guidance image or the like in this way, the route guidance device 10 may display a message or the like indicating that it is in the region where the display of the route guidance is specified to be stopped.

[0082] Further, the route guidance device 10 may display, together with or instead of this message, an area on the map image where the cancellation condition is satisfied. In this case, the route guidance device 10 may draw a polygon surrounded by the cancellation condition area defining points overlaid on the map image included in the map information to represent the cancellation condition area.

[0083] The administrator of the route guidance system 1 can set, for example, an area including the boarding and alighting positions of escalators, intersections, and other places where it is determined that the user of the route guidance device 1 should not focus on the screen as the cancellation condition area.

[0084] Note that the cancellation condition is not limited to the examples described here, and may be a condition based on the image data captured by the imaging unit 15, such as a condition that the image data captured by the imaging unit 15 includes a predetermined key image, or various conditions such as time zone, date, and day of the week can be adopted.

[0085] Furthermore, in the examples described here, while at least the display of the guidance image is cancelled while the cancellation condition is satisfied, the present embodiment is not limited to this. That is, when the route guidance device 10 of the present embodiment displays information including the guidance image, it may determine whether a predetermined implementation condition is satisfied, and display the guidance image only while the implementation condition is satisfied. In this example, when the implementation condition is not satisfied, at least the display of the guidance image is cancelled.

[0086] Also in this example, the implementation condition may be a condition based on the current position of the route guidance device 10 and a region (hereinafter referred to as the implementation condition region) predetermined in the map information. That is, the administrator of the route guidance system 1 may set the implementation condition region in the map information in advance. Here, the setting of the implementation condition region can also be performed by determining three or more points (implementation condition region defining points) surrounding the implementation condition region, similar to the cancellation condition region. The coordinates of these implementation condition region defining points are also assumed to be represented in the map coordinate system.

[0087] When the route guidance device 10 displays information including a guidance image, for example, when performing the processing of step S11 or step S21 in FIGS. 8(a) and 8(b), it checks whether the point CU in the map coordinate system representing the position of the route guidance device 10 at that time is inside the above implementation condition area. When the route guidance device 10 determines here that the point CU in the map coordinate system representing the position of the route guidance device 10 is not inside the above implementation condition area, it stops displaying at least a guidance image including a straight line representing an edge, an arrow representing the direction of the next node to pass through, and the like.

[0088] Also, when the route guidance device 10 determines here that the point CU in the map coordinate system representing the position of the route guidance device 10 is inside the above implementation condition area, it superimposes on the image data last captured by the imaging unit 15 and draws a guidance image including a virtual straight line representing an edge seen from the position CU, a virtual arrow representing the direction of the next node to pass through, and the like.

[0089] Note that when the route guidance device 10 determines that the point CU in the map coordinate system representing the position of the route guidance device 10 is not inside the above implementation condition area, it may stop displaying not only the guidance image but also the image data last captured by the imaging unit 15. Also, when stopping the display of the guidance image and the like in this way, the route guidance device 10 may display a message or the like indicating that it is in the area specified to stop the display of the route guidance.

[0090] Also, the route guidance device 10 may display, together with this message or instead of this message, an area on the map image where the implementation conditions are satisfied. At this time, the route guidance device 10 may draw a polygon surrounded by the implementation condition area specified points superimposed on the map image included in the map information to represent the implementation condition area.

[0091] The administrator of the route guidance system 1 can set, for example, an area including the vicinity of the guide board, the floor map, and other places where it is determined that there is no problem even if the user of the route guidance device 1 gazes at the screen as the implementation condition area.

[0092] Also, the implementation conditions are not limited to the conditions based on the area set on the map. As another example of the implementation conditions, for example, as a condition such as that the key image is included in the image data being captured by the imaging unit 15, it may be a condition determined based on the image data being captured by the imaging unit 15. Furthermore, conditions such as date and time and day of the week may also be used as the implementation conditions.

[0093] [Another example of the guidance image] In the description so far, the guidance image has been assumed to use virtual straight lines and figures that are superimposed and displayed on the image data being captured by the imaging unit 15, but the present embodiment is not limited to this. That is, the route guidance device 10 may display, as the guidance image, instead of or together with the virtual straight lines and figures that are superimposed and displayed on the image data being captured by the imaging unit 15, an image including a map image included in the map information and a marker representing the position of the route guidance device 10 on the map image.

[0094] [Modification example related to detection of key image] In the present embodiment, since the route guidance device 10 needs to identify the information on the position where the captured key image is posted, in the description so far, the key images have been assumed to be different from each other or to have a synthesized code image, but the present embodiment is not limited to this.

[0095] For example, when the route guidance device 10 can recognize the absolute direction with respect to the ground surface (for example, the north direction), that is, when it is equipped with a geomagnetic measuring instrument or the like, even if the posted key images are the same or similar in the image itself, if their posting directions (angles from the north direction) are different from each other, they can be identified respectively.

[0096] As illustrated in Fig. 9(a), using a point D in the direction of the key image (from a predetermined point B in the plane of the key image to the normal direction of the plane), a point C in the Z-axis (the Z-axis of the virtual space coordinate system) direction from the predetermined point B of the key image, and a northward N-axis, with the angle from the Z-axis of the N-axis being θ (where -π < θ ≤ π, with the clockwise direction being positive when viewing the floor from above), when taking a point n in the direction parallel to the N-axis from point B, subtracting the angle θ from the angle CBD gives the angle nBD.

[0097] Also, as illustrated in Fig. 9(b), using a point D' in the direction of the key image in the map coordinate system (from a point B' corresponding to the point B in the plane of the key image to the normal direction of the plane), a point C' in the Z'-axis (the Z'-axis of the map coordinate system) direction from the point B' of the key image, and a northward N'-axis, with the angle from the Z'-axis of the N'-axis being θ' (where -π < θ' ≤ π, with the clockwise direction being positive when viewing the floor from above), when taking a point n' (a point corresponding to the point n) in the direction parallel to the N'-axis from point B', the angle n'B'D' can be obtained by subtracting the angle C'B'n' = θ' from the angle C'B'D'.

[0098] In this example, in the map information, as a key image orientation feature amount, for each key image, the value of the above angle n'B'D' (this value is a known value that can be obtained in advance) is associated and recorded.

[0099] The route guidance device 10 obtains the angle nBD between the orientation of the key image (vector BD) detected from the captured image data and the northward orientation (vector BN) detected separately. Also, the route guidance device 10 searches for the key image included in the map information, which is associated with the image that matches the key image detected from the captured image data. Here, it is assumed that there are multiple identical or similar key images included in the map information, so as a result of this search, multiple key images (candidates) will be found.

[0100] The route guidance device 10 acquires, from the map information, the angle n'B'D' (hereinafter, in order to distinguish each, the angle n'B'D' for each candidate key image is referred to as angle αi (i = 1, 2,...; i corresponds to each candidate key image)) recorded in association with the candidate key images found by the search.

[0101] The route guidance device 10 obtains the absolute value βi = |αi - angle nBD| of the difference between the angle αi for each acquired candidate key image and the obtained angle nBD, and finds the minimum βi among these βi. Then, the route guidance device 10 identifies the key image associated with the angle αi corresponding to the minimum βi, and determines that it is the key image included in the image data in which the identified key image was captured. Then, the route guidance device 10 obtains information for identifying the determined key image.

[0102] The route guidance device 10 uses the information for identifying the key image obtained here, acquires the information on the position and orientation in the map coordinate system regarding the key image specified by the information for identifying the key image, and uses it for processes such as calculating the position of the route guidance device 10 in the map coordinate system.

[0103] [Floor identification] Also, as already described, in this embodiment, map information may be set across a plurality of floors. In this case, the route guidance device 10 needs to identify which floor it is located on.

[0104] Therefore, information for identifying the floor on which the key image is posted is included in the map information in association with the key image, and when the route guidance device 10 detects the key image from the captured image data, it may identify the floor on which it is located.

[0105] However, in this case, there is also a possibility that the route guidance device 10 may misidentify which floor it is located on immediately after moving between floors by an escalator, elevator, or the like. Therefore, key images may be posted on the floor surface, wall surface, etc. that are first imaged when reaching each floor, such as the exits of escalators or elevators for each floor.

[0106] In addition, when the route guidance device 10 incorporates a barometer, it may detect that the floor has been moved by using an air pressure difference or the like.

[0107] [Effects of Embodiment] According to the example of the present embodiment, when the route guidance device 10 detects an AR marker, which is a key image arranged at various locations in the facility, using virtual reality (AR) technology, the position and orientation of the AR marker in the AR coordinate system (which may coincide with, for example, the camera coordinate system) and the position and orientation of the corresponding AR marker in the map coordinate system fixed to the ground surface in advance as map information are used to specify the position and orientation of the route guidance device 10 in the map coordinate system.

[0108] Also, thereby, using the information of edges and nodes (represented in the map coordinate system) included in the network of passable routes set as map information, and using the information of the position and orientation of the route guidance device 10 in the specified map coordinate system, navigation by so-called navigation technology is realized. For this reason, it is possible to perform route guidance in various spaces including indoors.

[0109] Furthermore, in the present embodiment, using AR technology, on the screen of the route guidance device 10, a guidance image such as a figure of a straight line corresponding to an edge or a figure representing the direction of the next node to pass through is drawn and displayed by superimposing it on the image data of the passage in the facility that the route guidance device 10 is imaging. Thereby, it is possible to clearly indicate the moving direction to the user.

[0110] Then, the route guidance device 10 stops displaying the above guidance image within a region that satisfies the stop condition determined in advance by an administrator or the like. Alternatively, the guidance display is performed only within a region that satisfies the implementation condition determined in advance by an administrator or the like (so-called a zone where AR display is permitted).

[0111] Thereby, it is possible to suppress a decrease in the usage manner due to staring at the guidance display at an arbitrary location.

Explanation of Signs

[0112] 1 Route guidance system, 10 Route guidance device, 11 Control unit, 12 Memory unit, 13 Operation unit, 14 Display unit, 15 Imaging unit, 16 Communication unit, 20 Map information server, 21 Map information acquisition unit, 22 Key detection unit, 23 Relative position acquisition unit, 24 In-map position information acquisition unit, 25 Route guidance processing unit, 26 Information output unit.

Claims

1. A route guidance device for guiding a route formed by combining edges, wherein a predetermined key image is posted at a plurality of locations in the vicinity of the edge, map information defining the edge in a predetermined map coordinate system, and map information acquisition means for acquiring map information in which at least the positions where the key image is posted are defined by the map coordinate system, imaging means for imaging the periphery of the route guidance device and outputting the captured image data, key detection means for detecting the key image from the image data and outputting information on the position and orientation of the key image in a predetermined virtual space coordinate system, relative position acquisition means for acquiring relative position information between the information on the position and orientation of the key image in the virtual space coordinate system and the route guidance device based on the detected key image, and map internal position information acquisition means for acquiring the map internal position information of the route guidance device in the map coordinate system using the acquired map information and the acquired relative position information, wherein the map information includes information on the position and orientation of the key image defined by the map coordinate system, the map internal position information acquisition means acquires information on the distance between the key image and the route guidance device in the virtual space coordinate system and information on the angle of the route guidance device with respect to the orientation of the key image based on the acquired relative position information and the information on the position and orientation of the key image, and uses the size of the key image included in the captured image data, the information on the angle of the route guidance device with respect to the orientation of the acquired key image, and the information on the position and orientation of the key image defined in the map information to acquire the map internal position information of the route guidance device in the map coordinate system, and a route guidance device that uses the acquired map internal position information for a predetermined process of searching for and guiding a route along the edge.

2. The route guidance device according to claim 1, wherein the imaging means images the periphery of the route guidance device during route guidance and outputs the captured image data, the key detection means detects a key image from the image data, the relative position acquisition means acquires relative position information between the key image in a predetermined virtual space coordinate system and the route guidance device based on the detected key image when a key image is detected from the image data, and the map internal position information acquisition means acquires the map internal position information of the route guidance device in the map coordinate system of the route guidance device when the relative position information is acquired. The route guidance device is a route guidance device further comprising correction means for correcting information on the position of the route guidance device based on the acquired in-map position information. **Claim 3** The route guidance device according to claim 1 or 2, further comprising display means for superimposing and displaying an image captured by the imaging means and a guidance image for guiding a route during route guidance. **Claim 4** The route guidance device according to claim 3, wherein the display means suspends display of at least the guidance image while a predetermined suspension condition is satisfied. **Claim 5** The route guidance device according to claim 3, wherein the display means displays the guidance image only while a predetermined implementation condition is satisfied. **Claim 6** The route guidance device according to claim 4, wherein the suspension condition is a condition based on the position of the route guidance device and an area within the map, and the display means displays an area where the suspension condition is satisfied. **Claim 7** The route guidance device according to claim 5, wherein the implementation condition is a condition determined based on image data captured by the imaging means. **Claim 8** A key image posting body installed by an administrator and posting a predetermined key image at a plurality of positions near an edge where a user can pass, map information defining an edge in a predetermined map coordinate system, wherein at least information on the positions where the key images are posted is defined in the map coordinate system, and an information providing device providing the map information, and a route guidance system including a route guidance device carried by a user, wherein the route guidance device has map information acquisition means for acquiring the map information, imaging means for imaging the periphery of the route guidance device and outputting the captured image data, key detection means for detecting the key image from the image data and outputting information on the position and orientation of the key image in a predetermined virtual space coordinate system, relative position acquisition means for acquiring relative position information between the information on the position and orientation of the key image in the virtual space coordinate system and the route guidance device based on the detected key image, and in-map position information acquisition means for acquiring in-map position information of the route guidance device in the map coordinate system using the acquired map information and the acquired relative position information, wherein the map information includes information on the positions and orientations of the key images defined by the map coordinate system. The in-map position information acquisition means acquires, based on the acquired relative position information and the position and orientation information of the key image, the distance between the key image and the route guidance device in the virtual space coordinate system, and the information on the angle of the route guidance device with respect to the orientation of the key image, and uses the size of the key image included in the captured image data, the information on the angle of the route guidance device with respect to the orientation of the acquired key image, and the position and orientation information of the key image defined in the map information to acquire the in-map position information of the route guidance device in the map coordinate system. A route guidance system that uses the acquired in-map position information for a predetermined process of searching for and guiding a route along the edge. **Claim 9** A program that causes a computer to function as a route guidance device for guiding a route combined with edges, wherein a predetermined key image is posted at a plurality of locations near the edges. Map information that defines the edges by a predetermined map coordinate system, and map information acquisition means for acquiring map information in which at least the positions where the key images are posted are defined by the map coordinate system. Imaging means for imaging the periphery of a computer functioning as the route guidance device and outputting the captured image data. Key detection means for detecting the key image from the image data and outputting information on the position and orientation of the key image in a predetermined virtual space coordinate system. Relative position acquisition means for acquiring relative position information between the information on the position and orientation of the key image in the virtual space coordinate system and the computer functioning as the route guidance device based on the detected key image. In-map position information acquisition means for acquiring the in-map position information of the computer functioning as the route guidance device in the map coordinate system using the acquired map information and the acquired relative position information. Function as The map information includes information on the position and orientation of the key image defined by the map coordinate system. When causing the computer to function as the map internal position information acquisition means, based on the acquired relative position information, the position and orientation information of the key image, the computer causes the computer to acquire the distance between the key image in the virtual space coordinate system and the computer functioning as the route guidance device, and information on the angle of the computer functioning as the route guidance device with respect to the orientation of the key image. Using the size of the key image included in the captured image data, the information on the angle of the computer functioning as the route guidance device with respect to the orientation of the acquired key image, and the position and orientation information of the key image defined in the map information, the computer acquires the map internal position information of the computer functioning as the route guidance device in the map coordinate system, and provides the acquired map internal position information to a predetermined process for searching and guiding a route along the edge.

Images