Pedestrian device, mobile device, positioning system, and positioning method

Abstract

To improve efficiency in collation processing in visual positioning to shorten a processing time in the visual positioning.SOLUTION: A pedestrian terminal 1 includes: a camera 12 for photographing surroundings of a pedestrian; a state sensor 13 for detecting a state of the pedestrian; a memory 17 for storing a peripheral environment database DB related to a peripheral environment of the pedestrian; and a processor 18 for performing processing related to visual positioning in which an image captured by the camera is collated with the peripheral environment database DB to estimate a current position of the pedestrian. The processor comprises: measuring a gazing direction of the pedestrian on the basis of detection results of the state sensor; extracting, from the peripheral environment DB, this gazing direction as an optimum collation direction having high possibility that collation is successful and a candidate image corresponding to the gazing direction as a collation object; collating the image captured by the camera with the candidate image; and obtaining positional information associated with the candidate image, where the collation is successful, as current positional information of the pedestrian.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a pedestrian device that is held by a pedestrian to measure the current position of the pedestrian, a mobile device that is held by a mobile body such as a pedestrian or a vehicle to measure the current position of the mobile body, a positioning system, and a positioning method. [Background technology]

[0002] In a safe driving support wireless system that uses ITS (Intelligent Transport System), vehicle location information is exchanged between in-vehicle terminals to avoid accidents between vehicles, and vehicle-pedestrian accidents are avoided by exchanging vehicle and pedestrian location information between in-vehicle terminals and pedestrian terminals.

[0003] In-vehicle and pedestrian terminals mainly obtain the position information of vehicles and pedestrians by satellite positioning, but various positioning methods can also be adopted, such as positioning using Pedestrian Dead Reckoning (PDR). In this case, it is desirable to adopt high-precision positioning technology in order to prevent traffic accidents.

[0004] One such highly accurate positioning technology is known as visual positioning, for example, a technology called VPS (Visual Positioning Service) (see Patent Document 1). In this technology, a positioning device compares a captured image output from a camera in real time with a candidate image registered in a database, and if the two match, acquires the location information associated with the candidate image as the location information of the current location of the moving object. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2012 / 086821 Summary of the Invention [Problem to be solved by the invention]

[0006] With satellite positioning, the accuracy of positioning decreases due to the effects of multipath and the blockage of satellite radio waves caused by pedestrians entering the shadows of buildings. On the other hand, visual positioning can ensure high accuracy, but has the problem of taking a long time to obtain positioning results due to the heavy load of the matching process. For this reason, technology to shorten the processing time of visual positioning is desired.

[0007] Therefore, the main object of the present invention is to provide a pedestrian device, a mobile device, a positioning system, and a positioning method that can improve the efficiency of the matching process in visual positioning in order to shorten the processing time of visual positioning. [Means for solving the problem]

[0008] The pedestrian device of the present invention comprises an external sensor that detects objects around the pedestrian, a memory that stores ambient environment information about the pedestrian's surroundings, and a processor that performs processing related to visual positioning that estimates the pedestrian's current position by comparing an image detected by the external sensor with the ambient environment information. The processor estimates an optimal matching direction that is likely to result in successful matching, and in the visual positioning, extracts candidate information corresponding to the optimal matching direction from the ambient environment information as a matching target, compares the image detected by the external sensor with the candidate information, and obtains location information associated with the candidate information that has been successfully matched as the pedestrian's current location information.

[0009] In addition, the mobile device of the present invention comprises an external sensor that detects objects around the mobile device, a memory that stores ambient environment information regarding the mobile device's surroundings, and a processor that performs processing related to visual positioning, which estimates the current position of the mobile device by matching the image detected by the external sensor with the ambient environment information, and is configured to estimate an optimal matching direction that is likely to result in successful matching, extract candidate information corresponding to the optimal matching direction from the ambient environment information as a matching target in the visual positioning, match the image detected by the external sensor with the candidate information, and obtain location information associated with the candidate information for which matching was successful as the current location information of the mobile device.

[0010] In addition, the positioning system of the present invention is a positioning system composed of one or more computers that execute processing to obtain position information of a mobile body in a mobile body device, and is equipped with an external sensor that is installed in the mobile body device and detects objects around the mobile body, and the computer includes a memory that stores ambient environment information regarding the mobile body's ambient environment, and a processor that performs processing related to visual positioning that estimates the current position of the mobile body by matching the image detected by the external sensor with the ambient environment information, and the processor is configured to estimate an optimal matching direction that is likely to result in successful matching, and in the visual positioning, extract candidate information corresponding to the optimal matching direction from the ambient environment information as a matching target, match the image detected by the external sensor with the candidate information, and obtain the position information associated with the candidate information that has been successfully matched as the current position information of the mobile body.

[0011] Furthermore, the positioning method of the present invention is a positioning method in which one or more computers execute a process for acquiring position information of a mobile body in a mobile body device, and the computer performs a process for visual positioning that estimates the current position of the mobile body by comparing an image detected by an external sensor that detects objects around the mobile body with surrounding environment information related to the mobile body's surrounding environment, and estimates an optimal matching direction that is likely to result in successful matching.In the visual positioning, candidate information corresponding to the optimal matching direction is extracted from the surrounding environment information as a matching target, the image detected by the external sensor is compared with the candidate information, and the position information associated with the candidate information for which matching was successful is acquired as the current position information of the mobile body. [Effects of the Invention]

[0012] According to the present invention, the processing time of visual positioning can be reduced, and therefore the efficiency of the matching process in visual positioning can be improved. [Brief explanation of the drawings]

[0013] [Figure 1] Overall configuration diagram of the traffic safety support system according to the first embodiment [Figure 2] FIG. 1 is an explanatory diagram showing an overview of visual positioning performed by a pedestrian terminal according to a first embodiment; [Figure 3] FIG. 10 is an explanatory diagram showing an overview of a match candidate extraction process performed by a pedestrian terminal according to the first embodiment; [Figure 4] FIG. 1 is a block diagram showing a schematic configuration of a pedestrian terminal according to a first embodiment; [Figure 5] FIG. 1 is a block diagram showing a schematic configuration of a roadside device according to a first embodiment. [Figure 6] FIG. 1 is a flowchart showing an operation procedure of a pedestrian terminal according to a first embodiment; [Figure 7] FIG. 1 is a flowchart showing an operation procedure of a pedestrian terminal according to a first embodiment; [Figure 8] FIG. 1 is a flowchart showing an operation procedure of the in-vehicle terminal according to the first embodiment; [Figure 9] FIG. 1 is a flowchart showing an operation procedure of a roadside device according to a first embodiment; [Figure 10] FIG. 10 is an explanatory diagram showing an overview of the foot image matching process performed by the pedestrian terminal according to the second embodiment; [Figure 11] FIG. 10 is a block diagram showing a schematic configuration of a pedestrian terminal according to a second embodiment. [Figure 12] FIG. 11 is an explanatory diagram showing an overview of control performed by a pedestrian terminal according to a third embodiment; [Figure 13] FIG. 10 is a block diagram showing a schematic configuration of a pedestrian terminal according to a third embodiment. [Figure 14] FIG. 10 is a block diagram showing a schematic configuration of a roadside device according to a third embodiment. [Figure 15] FIG. 10 is a flowchart showing the operation procedure of the pedestrian terminal according to the third embodiment. [Figure 16] FIG. 10 is an explanatory diagram showing an overview of control performed by a pedestrian terminal according to a fourth embodiment; [Figure 17] FIG. 10 is a block diagram showing a schematic configuration of a pedestrian terminal according to a fourth embodiment. [Figure 18] FIG. 10 is a flowchart showing the operation procedure of the pedestrian terminal according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] The first invention made to solve the above problem comprises an external sensor that detects objects around a pedestrian, a memory that stores ambient environment information about the pedestrian's surroundings, and a processor that performs processing related to visual positioning that estimates the pedestrian's current location by matching an image detected by the external sensor with the ambient environment information, wherein the processor estimates an optimal matching direction that is likely to result in successful matching, and in the visual positioning, extracts candidate information corresponding to the optimal matching direction from the ambient environment information as a matching target, matches the image detected by the external sensor with the candidate information, and obtains location information associated with the candidate information that has been successfully matched as the pedestrian's current location information.

[0015] According to this method, the direction where matching is likely to be successful, i.e., the direction where the most suitable candidate information exists, is estimated as the optimal matching direction, and candidate information corresponding to the optimal matching direction is extracted from the surrounding environment information as a matching target. This improves the efficiency of the matching process in visual positioning.

[0016] In addition, a second invention is configured to include a status sensor that detects the status of a pedestrian, and the processor measures the pedestrian's gaze direction based on the detection result of the status sensor, and extracts the candidate information corresponding to the gaze direction as the optimal matching direction from the surrounding environment information as a matching target.

[0017] This allows candidate information corresponding to the pedestrian's gaze direction to be matched with the image detected by the external sensor, thereby improving the efficiency of the matching process in visual positioning.

[0018] The third invention is configured as a wearable device attached to the body of a pedestrian, wherein the state sensor detects at least one of the pedestrian's head direction and gaze direction, and the processor measures the gaze direction based on at least one of the head direction and gaze direction.

[0019] This makes it possible to measure the gaze direction of a pedestrian with high accuracy.

[0020] In a fourth aspect of the present invention, the processor is configured to extract, from the surrounding environment information, the candidate information including a feature point located in the gaze direction as a matching target.

[0021] This allows candidate information including feature points located in the pedestrian's gaze direction to be matched with the image detected by the external sensor, thereby improving the efficiency of the matching process in visual positioning.

[0022] In addition, a fifth invention is configured such that the external sensor is a camera that captures images of the pedestrian's surroundings, and the processor, in the visual positioning, compares the captured image of the pedestrian's surroundings captured by the camera with the candidate information.

[0023] This allows for efficient visual positioning over a wide area.

[0024] In addition, a sixth invention is configured such that the external sensor is a camera that photographs the feet of a pedestrian, and the processor compares an image of the feet of the pedestrian photographed by the camera with the candidate information.

[0025] This makes it possible to obtain highly accurate positioning results.

[0026] In a seventh aspect of the present invention, the memory stores feature information including position information of features that serve as indicators on a map, and the processor selects a feature of interest from features present around the pedestrian based on the pedestrian's position information acquired by standard positioning and the feature information, acquires a direction in which the feature of interest exists relative to the pedestrian, and extracts the candidate information including the feature of interest from the surrounding environment information based on the direction in which the feature of interest exists as the optimal matching direction in the visual positioning. Note that standard positioning is radio wave positioning that acquires position information based on incoming radio waves, such as satellite positioning that acquires position information based on radio waves from a positioning satellite.

[0027] This allows candidate information including a feature of interest to be matched with an image detected by an external sensor, thereby improving the efficiency of the matching process in visual positioning.

[0028] In addition, an eighth invention is configured such that the processor determines whether the target feature is included in the detection range of the external sensor based on the direction in which the target feature is located, and if the target feature is not included in the detection range of the external sensor, controls the detection direction of the external sensor so that the target feature is included in the detection range of the external sensor.

[0029] This makes it possible to acquire an image detected by an external sensor that includes a feature of interest, thereby improving the efficiency of the matching process in visual positioning.

[0030] In addition, a ninth invention is configured such that the processor determines whether the target feature is included in the detection range of the external sensor based on the direction in which the target feature is located, and if the target feature is not included in the detection range of the external sensor, performs guidance control to encourage the pedestrian to take action to change the detection direction of the external sensor so that the target feature is included in the detection range of the external sensor.

[0031] This allows the acquisition of an image detected by an external sensor that includes a target feature. This improves the efficiency of the matching process in visual positioning. The guidance control includes, for example, displaying a guidance screen on a display or outputting a guidance voice from a speaker.

[0032] In addition, a tenth aspect of the present invention is configured such that the feature information includes information on lighting devices installed along the road as the feature.

[0033] This makes it possible to improve the efficiency of the matching process in visual positioning even at night.

[0034] An eleventh invention includes an external sensor that detects objects around a moving body, a memory that stores ambient environment information about the surrounding environment of the moving body, and a processor that performs processing related to visual positioning that estimates the current position of the moving body by matching an image detected by the external sensor with the ambient environment information, wherein the processor estimates an optimal matching direction that is likely to result in successful matching, and in the visual positioning, extracts candidate information corresponding to the optimal matching direction from the ambient environment information as a matching target, matches the image detected by the external sensor with the candidate information, and obtains location information associated with the candidate information that has been successfully matched as the current location information of the moving body.

[0035] According to this, as in the first aspect of the invention, the processing time of visual positioning can be shortened, and therefore the efficiency of the matching process in visual positioning can be improved.

[0036] In addition, a twelfth invention is a positioning system consisting of one or more computers that execute a process to obtain position information of a mobile body in a mobile body device, the system comprising: an external sensor provided in the mobile body device and detecting objects around the mobile body; a memory that stores ambient environment information regarding the mobile body's ambient environment; and a processor that performs processing related to visual positioning, which estimates the current position of the mobile body by matching an image detected by the external sensor with the ambient environment information, wherein the processor estimates an optimal matching direction that is likely to result in successful matching, and in the visual positioning, extracts candidate information corresponding to the optimal matching direction from the ambient environment information as a matching target, matches the image detected by the external sensor with the candidate information, and obtains the position information associated with the candidate information for which matching was successful as the current position information of the mobile body.

[0037] According to this, as in the first aspect of the invention, the processing time of visual positioning can be shortened, and therefore the efficiency of the matching process in visual positioning can be improved.

[0038] In addition, a thirteenth invention is a positioning method in which one or more computers execute a process for acquiring position information of a mobile body in a mobile body device, wherein the computer performs a process for visual positioning that estimates the current position of the mobile body by comparing an image detected by an external sensor that detects objects around the mobile body with surrounding environment information related to the mobile body's surrounding environment, estimates an optimal matching direction that is likely to result in successful matching, extracts candidate information corresponding to the optimal matching direction from the surrounding environment information as a matching target, compares the image detected by the external sensor with the candidate information, and acquires the position information associated with the candidate information for which matching was successful as the current position information of the mobile body.

[0039] According to this, as in the first aspect of the invention, the processing time of visual positioning can be shortened, and therefore the efficiency of the matching process in visual positioning can be improved.

[0040] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0041] (First embodiment) FIG. 1 is a diagram showing the overall configuration of a traffic safety support system according to the first embodiment.

[0042] This traffic safety support system supports traffic safety for pedestrians and vehicles, and includes a pedestrian terminal 1 (pedestrian device, mobile device, computer), an in-vehicle terminal 2 (in-vehicle device), and a roadside unit 3 (roadside device).

[0043] ITS communication is performed between the pedestrian terminal 1, the in-vehicle terminal 2, and the roadside unit 3. This ITS communication is wireless communication using a frequency band (e.g., 700 MHz band or 5.8 GHz band) adopted in a safe driving support wireless system that uses an ITS (Intelligent Transport System). In this embodiment, ITS communication between the pedestrian terminal 1 and the in-vehicle terminal 2 will be referred to as pedestrian-to-vehicle communication, ITS communication between the pedestrian terminal 1 and the roadside unit 3 will be referred to as road-to-pedestrian communication, and ITS communication between the in-vehicle terminal 2 and the roadside unit 3 will be referred to as road-to-vehicle communication. ITS communication is also performed between in-vehicle terminals 2, and this ITS communication will be referred to as vehicle-to-vehicle communication.

[0044] The pedestrian terminal 1 is carried by a pedestrian. This pedestrian terminal 1 sends and receives messages including location information and the like to an in-vehicle terminal 2 via ITS communication (pedestrian-to-vehicle communication), determines the risk of collision between the pedestrian and a vehicle, and if there is a risk of collision, activates an action to warn the pedestrian. The pedestrian terminal 1 is a wearable device attached to the body of the pedestrian, particularly a wearable device attached to the head of the pedestrian, a device known as a head-mounted device or smart glasses, and is equipped with a function to realize AR (Augmented Reality). Note that the wearable device serving as the pedestrian terminal 1 does not necessarily have to be attached to the head of the pedestrian, as long as it can detect the head direction or line of sight of the pedestrian, and any form of attachment is acceptable.

[0045] The in-vehicle terminal 2 is mounted on a vehicle. This in-vehicle terminal 2 transmits and receives messages including location information and the like to and from the pedestrian terminal 1 via ITS communication (pedestrian-to-vehicle communication), determines the risk of collision between the pedestrian and the vehicle, and performs an attention-calling operation for the driver if there is a risk of collision. The attention-calling operation may be performed using, for example, a car navigation device.

[0046] The roadside unit 3 is installed at road intersections, etc. The roadside unit 3 distributes various information such as traffic information to the pedestrian terminal 1 and the in-vehicle terminal 2 through ITS communication (road-to-pedestrian communication, road-to-vehicle communication). The roadside unit 3 also notifies the in-vehicle terminal 2 and the pedestrian terminal 1 of the presence of vehicles and pedestrians located around the unit through ITS communication (road-to-vehicle communication, road-to-pedestrian communication). This makes it possible to prevent collisions at intersections where there is no line of sight.

[0047] Here, the pedestrian terminal 1 constantly performs satellite positioning as standard positioning to acquire position information of the pedestrian's current location. Satellite positioning acquires position information based on radio waves from positioning satellites. Note that standard positioning may also be radio wave positioning that acquires position information based on incoming radio waves, for example, positioning that acquires position information based on radio waves from a terrestrial base station. Standard positioning may also be positioning using PDR (pedestrian dead-reckoning). In this PDR positioning, the pedestrian terminal 1 estimates the relative movement amount of the pedestrian from the previous position based on detection results from a gyro sensor, acceleration sensor, etc., and relatively estimates the pedestrian's current position.

[0048] Next, the visual positioning performed by the pedestrian terminal 1 according to the first embodiment will be described. FIG.

[0049] The pedestrian terminal 1 is equipped with a camera 12 (see FIG. 4) that captures images of the pedestrian's surroundings. The pedestrian terminal 1 performs visual positioning to acquire location information of the pedestrian's current location using images captured by the camera 12. In visual positioning, the captured images output in real time from the camera 12 (hereinafter referred to as "real-time captured images") are compared with candidate images registered in a surrounding environment DB (surrounding environment database), and location information corresponding to the candidate image that has been successfully matched is acquired as location information of the pedestrian's current location.

[0050] The surrounding environment DB stores information about the surrounding environment of a pedestrian (surrounding environment information). Specifically, an image captured on a road where a pedestrian is walking, in a direction that corresponds to the line of sight of the pedestrian looking ahead, is associated with the location information of the capture point and registered in the surrounding environment DB. Note that instead of the captured image itself, feature information (information on feature points) extracted from the captured image may also be registered in the surrounding environment DB.

[0051] In visual positioning, captured images registered in the surrounding environment DB are extracted as candidate images, and if the candidate image corresponds to the shooting direction of camera 12, i.e., corresponds to the view that the pedestrian sees from the pedestrian's current location, it is successfully matched with the captured image of camera 12.

[0052] The surrounding environment DB may also store three-dimensional map information (environmental map information). The three-dimensional map information is three-dimensional information about objects present in the scenery visible to pedestrians, specifically, roads and fixed structures such as buildings in the vicinity. This three-dimensional map information is generated based on, for example, images captured from various directions of a target space including roads on which pedestrians pass.

[0053] In this case, the information extracted from the surrounding environment DB as the matching target is a set of feature points, and this set of feature points is compared as candidate information with the feature points extracted from the image captured by the camera 12 in the matching process.

[0054] In this embodiment, a configuration in which candidate images are extracted from the surrounding environment DB will be mainly described, but this description can also be interpreted as a configuration in which candidate information as a set of feature points is extracted from the surrounding environment DB. Furthermore, the term "candidate information" is used to represent both the candidate image and the candidate information as a set of feature points, as appropriate.

[0055] The roadside device 3 stores registration information of the surrounding environment DB relating to a predetermined area around the roadside device 3. When the roadside device 3 approaches the pedestrian terminal 1, the roadside device 3 distributes the registration information of the surrounding environment DB of the roadside device 3 to the pedestrian terminal 1.

[0056] Furthermore, in this embodiment, visual positioning is performed by the pedestrian terminal 1, but visual positioning may also be performed by the vehicle-mounted terminal 2.

[0057] Next, a description will be given of the matching candidate extraction process performed by the pedestrian terminal 1 according to the first embodiment. FIG.

[0058] The pedestrian terminal 1 performs visual positioning. In visual positioning, candidate images are extracted from the surrounding environment DB, and the candidate images are compared with the images captured by the camera 12. At this time, candidate images corresponding to the surroundings of the pedestrian's current position are extracted in order. At this time, candidate images are first extracted based on the pedestrian's traveling direction. Specifically, candidate images located ahead in the pedestrian's traveling direction are extracted.

[0059] In this embodiment, the pedestrian terminal 1 is configured as a wearable device attached to the head of the pedestrian, and the camera 12 is provided so as to face the front of the pedestrian. Therefore, when the pedestrian is walking while looking straight ahead, a candidate image located ahead in the pedestrian's traveling direction is extracted, thereby obtaining a candidate image corresponding to the capture range of the image captured by the camera 12, and the matching process is completed quickly.

[0060] On the other hand, when a pedestrian changes his / her direction of travel, immediately before changing his / her direction of travel, the pedestrian gazes at an object located in the new direction of travel. In this case, the pedestrian's direction of travel and the pedestrian's gaze direction, i.e., the shooting direction of camera 12, are misaligned. Therefore, if a candidate image is extracted based on the pedestrian's direction of travel, the extracted candidate image does not correspond to the shooting range of the image captured by camera 12, and the matching process takes time.

[0061] Therefore, in this embodiment, the pedestrian terminal 1 is equipped with a state sensor 13 (see FIG. 4) that detects the state of the pedestrian, and measures the pedestrian's gaze direction based on the detection result of the state sensor 13. Candidate information corresponding to the gaze direction is extracted from the surrounding environment DB as a matching target. Specifically, candidate information including feature points located in the pedestrian's gaze direction is extracted from the surrounding environment DB as a matching target. The example shown in FIG. 3 is a case where a pedestrian turns left at the next intersection, and the pedestrian gazes diagonally ahead to the left. Therefore, a candidate image located diagonally ahead of the pedestrian and to the left is extracted from the surrounding environment DB, rather than a candidate image located ahead in the pedestrian's traveling direction.

[0062] In this embodiment, the pedestrian terminal 1 is configured as a wearable device attached to the head of the pedestrian, and the state sensor 13 detects the head direction (face direction) of the pedestrian and the gaze direction (point of view) based on the head of the pedestrian, and the gaze direction of the pedestrian is measured based on the head direction and gaze direction. Note that the head direction may be used as the gaze direction without measuring the gaze direction.

[0063] Note that positioning accuracy improves by recognizing as many feature points as possible around the pedestrian. Furthermore, in locations without distinctive objects, it is difficult to acquire a sufficient number of appropriate feature points, resulting in a decrease in positioning accuracy. However, in locations with distinctive objects, positioning accuracy improves. Meanwhile, pedestrians may gaze in a direction different from their direction of travel just before taking some action. Furthermore, in unfamiliar locations, pedestrians often look around while walking. Furthermore, if there is a distinctive object around the pedestrian, the pedestrian will gaze at that distinctive object. Therefore, not only when the pedestrian changes their direction of travel, but also in situations where the pedestrian is walking while looking around, candidate images can be extracted based on the gaze direction, thereby improving the efficiency of the matching process in visual positioning and increasing the positioning accuracy.

[0064] Next, a description will be given of the schematic configuration of the pedestrian terminal 1 according to the first embodiment. Fig. 4 is a block diagram showing the schematic configuration of the pedestrian terminal 1.

[0065] The pedestrian terminal 1 includes a satellite positioning unit 11, a camera 12 (external environment sensor), a status sensor 13, an AR display 14, an ITS communication unit 15, a wireless communication unit 16, a memory 17, and a processor 18.

[0066] Satellite positioning unit 11 measures the position of the device itself using a satellite positioning system such as GPS (Global Positioning System) or QZSS (Quasi-Zenith Satellite System), and acquires position information (latitude and longitude) of the device itself.

[0067] The camera 12 captures an image of the area in front of the pedestrian.

[0068] The state sensor 13 detects the state of the pedestrian. In this embodiment, the pedestrian terminal 1 is equipped with an acceleration sensor 21, a gyro sensor 22 (angular velocity sensor), a geomagnetic sensor 23, and a gaze sensor 24 as the state sensor 13. The acceleration sensor 21 detects the acceleration occurring in the body of the pedestrian. The gyro sensor 22 detects the angular velocity occurring in the body of the pedestrian. The geomagnetic sensor 23 detects the geomagnetic direction. The gaze sensor 24 (gaze camera) captures the left and right eyeballs of the pedestrian.

[0069] The AR display 14 superimposes and displays a virtual object on a real space within the pedestrian's field of vision, thereby realizing AR (Augmented Reality).

[0070] The ITS communication unit 15 broadcasts messages to the in-vehicle terminal 2 and the roadside unit 3 through ITS communication (pedestrian-to-vehicle communication and road-to-pedestrian communication), and also receives messages transmitted from the in-vehicle terminal 2 and the roadside unit 3.

[0071] The wireless communication unit 16 transmits messages to the roadside unit 3 and receives messages transmitted from the roadside unit 3 by wireless communication such as WiFi (registered trademark).

[0072] The memory 17 stores map information, programs executed by the processor 18, and the like. The memory 17 also stores registration information (candidate images and location information) in the surrounding environment DB. In this embodiment, when the pedestrian terminal 1 approaches an intersection, it acquires registration information in the surrounding environment DB related to the area surrounding the intersection from the roadside unit 3 installed at the intersection. In addition, feature information (information on a plurality of feature points) extracted from the candidate image may be registered in the surrounding environment DB, rather than the candidate image itself.

[0073] The processor 18 performs various processes by executing programs stored in the memory 17. In this embodiment, the processor 18 performs a message control process P1, a collision determination process P2, an attention-call control process P3, a status information acquisition process P4, and a visual positioning process P5.

[0074] In the message control process P1, the processor 18 controls the transmission and reception of messages of ITS communication between the in-vehicle terminal 2 and the roadside device 3. The processor 18 also controls the transmission and reception of messages of wireless communication with the roadside device 3.

[0075] In the collision determination process P2, the processor 18 determines whether there is a risk of the vehicle colliding with a pedestrian based on the vehicle position information contained in the vehicle information obtained from the in-vehicle terminal 2 and the pedestrian position information obtained by the satellite positioning unit 11.

[0076] In the attention-calling control process P3, the processor 18 performs control so as to perform a predetermined attention-calling action (such as audio output or vibration) for the pedestrian when it is determined in the collision determination process P2 that there is a risk of collision.

[0077] In the state information acquisition process P4, the processor 18 acquires state information representing the state of the pedestrian based on the detection result of the state sensor 13. In this state information acquisition process P4, a speed measurement process P11, a direction measurement process P12, and a line of sight measurement process P13 are performed.

[0078] In the speed measurement process P11, the processor 18 measures the moving speed of the pedestrian based on the detection result of the acceleration sensor 21. When the pedestrian walks, acceleration occurs in the body of the pedestrian, and the walking pitch of the pedestrian is calculated based on the change in this acceleration. The speed is calculated from the walking pitch and stride length. The stride length may be set based on the attributes of the pedestrian (adult, child, etc.) registered in the pedestrian terminal 1.

[0079] In the direction measurement process P12, the processor 18 measures the head direction (face direction) of the pedestrian wearing the pedestrian terminal 1 and the moving direction of the pedestrian, based on the detection results of the acceleration sensor 21, the gyro sensor 22, and the geomagnetic sensor 23. The processor 18 also measures the gaze direction of the pedestrian, based on the gaze direction and head direction of the pedestrian acquired in the gaze measurement process P13.

[0080] In gaze measurement processing P13, processor 18 measures the gaze direction (viewpoint) of the pedestrian (gaze sensing) based on the detection result (captured image) of gaze sensor 24. Specifically, it acquires position information of the pedestrian's viewpoint, i.e., the coordinate value of the viewpoint in the coordinate system of the pedestrian's field of view.

[0081] In the visual positioning process P5, the processor 18 compares the captured image output in real time from the camera 12 with candidate images extracted from the surrounding environment DB of the host device to estimate the current position of the pedestrian. In this visual positioning process P5, a comparison target extraction process P15, an image comparison process P16, and a position information acquisition process P17 are performed.

[0082] In the matching target extraction process P15, the processor 18 extracts candidate images (candidate information) to be matched from the surrounding environment DB of the host device based on the pedestrian's position information acquired by satellite positioning and the pedestrian's status information acquired in the status information acquisition process P4. In this embodiment, candidate images corresponding to the pedestrian's gaze direction acquired in the direction measurement process P12 are extracted from the surrounding environment DB. Specifically, candidate images including feature points located in the pedestrian's gaze direction are extracted from the surrounding environment DB.

[0083] In the image matching process P16, the processor 18 matches the captured image output in real time from the camera 12 with the candidate image extracted in the matching target extraction process P15. At this time, the processor 18 extracts feature information (information on a plurality of feature points) from each of the captured image and the candidate image, and compares the feature information to perform image matching.

[0084] In the position information acquisition process P17, the processor 18 acquires the position information associated with the candidate image that has been successfully matched in the image matching process P16 as the position information of the pedestrian's current location.

[0085] The in-vehicle terminal 2 also includes a processor and memory (not shown) and can perform message control processing, collision determination processing, and attention-call control processing by executing programs stored in the memory.

[0086] In this embodiment, the pedestrian terminal 1 includes a camera 12 as an external sensor that detects objects around the pedestrian. However, the external sensor is not limited to the camera 12. For example, the external sensor may be a LIDAR (Light Detection and Ranging) sensor that detects objects using laser light. Alternatively, the external sensor may be a radar that detects objects using radio waves. Alternatively, the external sensor may be a sensor having a depth prediction function (a depth measurement function) that measures depth, i.e., the distance (depth) from the external sensor to an object. In visual positioning, a detected image output in real time from such an external sensor is compared with a candidate image registered in the surrounding environment DB. Note that the image in this embodiment may include not only two-dimensional information but also three-dimensional information, such as a distance image acquired by a depth camera. Furthermore, the image matching process P10 may be performed by matching two pieces of two-dimensional information together, or by matching two pieces of three-dimensional information together, or by matching two pieces of two-dimensional information together with three-dimensional information.

[0087] In addition, in this embodiment, the camera 12 is configured to capture an image in front of the pedestrian, but the camera 12 may be configured to capture an image in any direction around the pedestrian, for example, behind the pedestrian, or to capture an image in a wider range around the pedestrian.

[0088] Furthermore, in this embodiment, the pedestrian terminal 1 is configured as a wearable device, particularly a wearable device worn on the head of the pedestrian, but the pedestrian terminal 1 may also be configured as a wearable device and a mobile terminal (main body). In this case, the camera 12, status sensor 13, AR display 14, etc. may be provided in the wearable device, and the processor 18, memory 17, etc. may be provided in the mobile terminal. Furthermore, the pedestrian terminal 1 may not be a wearable device, but may be configured as a mobile terminal such as a smartphone.

[0089] Furthermore, the functions of the pedestrian terminal 1 may be provided in a cloud computer. For example, in this embodiment, the pedestrian terminal 1 performs processes such as visual positioning, but this process may be performed in a cloud computer. In this case, the camera 12, the status sensor 13, the AR display 14, and the like are provided in the pedestrian terminal 1.

[0090] Next, a description will be given of the general configuration of the roadside unit 3. FIG.

[0091] The roadside unit 3 includes an ITS communication unit 31, a wireless communication unit 32, a memory 33, and a processor .

[0092] The ITS communication unit 31 broadcasts messages to the pedestrian terminal 1 and the in-vehicle terminal 2 through ITS communication (road-to-pedestrian communication, road-to-vehicle communication), and also receives messages transmitted from the pedestrian terminal 1 and the in-vehicle terminal 2.

[0093] The wireless communication unit 32 transmits messages to the pedestrian terminal 1 and receives messages transmitted from the pedestrian terminal 1 via wireless communication such as WiFi (registered trademark).

[0094] The memory 33 stores programs executed by the processor 34. In this embodiment, the memory 33 also stores registration information of the surrounding environment DB.

[0095] The processor 34 performs various processes by executing programs stored in the memory 33. In this embodiment, the processor 34 performs message control processing and surrounding environment DB management processing.

[0096] In the message control process, the processor 34 controls the transmission and reception of messages of ITS communication between the pedestrian terminal 1 and the in-vehicle terminal 2. The processor 34 also controls the transmission and reception of messages of wireless communication with the pedestrian terminal 1.

[0097] In the surrounding environment DB management process, the processor 34 manages the surrounding environment DB. Specifically, in response to a request from the pedestrian terminal 1, the processor 34 distributes registration information of the surrounding environment DB from the wireless communication unit 32 to the pedestrian terminal 1.

[0098] Incidentally, the functions of the roadside device 3 may be provided in a cloud computer. For example, in the present embodiment, the surrounding environment DB is provided in the roadside device 3, but the surrounding environment DB may also be provided in a cloud computer. In this case, the pedestrian terminal 1 may be configured to download the registration information of the surrounding environment DB by communicating with the cloud computer via the roadside device 3. Furthermore, the pedestrian terminal 1 may be configured to have a high-speed cellular communication function such as 5G and to download the registration information of the surrounding environment DB by communicating with the cloud computer via a cellular communication network or the like. In this case, the pedestrian terminal 1 may be configured to notify the cloud computer of its own location information, and the cloud computer may distribute the registration information of the surrounding environment DB for a required range around the pedestrian terminal 1 to the pedestrian terminal 1.

[0099] Next, the operation procedures of the pedestrian terminal 1, the in-vehicle terminal 2, and the roadside device 3 according to the first embodiment will be described. FIGS. 6 and 7 are flow diagrams showing the operation procedures of the pedestrian terminal 1. FIG. 8 is a flow diagram showing the operation procedures of the in-vehicle terminal 2. FIG. 9 is a flow diagram showing the operation procedures of the roadside device 3. Note that each process shown in FIGS. 6(A), (B), (C), FIG. 8, and FIGS. 9(A) and (B) is performed at predetermined intervals. In other words, these processes are repeatedly executed even after they have been completed.

[0100] 6(A), in the pedestrian terminal 1, first, the satellite positioning unit 11 acquires the position information of the pedestrian (ST101). Next, the processor 18 determines, based on the position information of the pedestrian, whether or not the situation requires the pedestrian information to be transmitted, specifically, whether or not the pedestrian has entered a dangerous area (for example, an intersection) (ST102).

[0101] Here, if the situation requires the pedestrian information to be transmitted (Yes in ST102), the ITS communication unit 15 transmits an ITS communication message including the pedestrian information (such as the pedestrian ID and location information) to the in-vehicle terminal 2 and the roadside unit 3 in response to a transmission instruction from the processor 18 (ST103).

[0102] As shown in FIG. 8, when the in-vehicle terminal 2 receives an ITS communication (pedestrian-to-vehicle communication) message from the pedestrian terminal 1 (Yes in ST201), it performs a collision determination to determine whether there is a risk of the vehicle colliding with a pedestrian based on the vehicle position information contained in the message (ST202).

[0103] If there is a risk of the host vehicle colliding with a pedestrian (Yes in ST202), a predetermined attention-calling operation is performed for the driver (ST203). Specifically, as the attention-calling operation, the car navigation device is made to perform an attention-calling operation (for example, audio output or screen display). If the host vehicle is an autonomous vehicle, the autonomous driving ECU (cruise control device) is instructed to perform a predetermined collision avoidance operation.

[0104] 9(A), in the roadside device 3, when the ITS communication unit 31 receives a message of ITS communication (pedestrian-to-vehicle communication) from the pedestrian terminal 1 (Yes in ST301), the processor 34 acquires the terminal ID and location information of the pedestrian terminal 1 included in the received message (ST302). Next, based on the pedestrian's location information, the processor 34 determines whether the pedestrian terminal 1 is located in the vicinity of (inside or near) the target area of ​​the registered information in the surrounding environment DB (ST303).

[0105] Here, if the pedestrian terminal 1 is located in the vicinity of the target area (Yes in ST303), in response to a transmission instruction from the processor 34, the ITS communication unit 31 transmits to the pedestrian terminal 1 an ITS communication message including DB usage information indicating that the pedestrian terminal 1 can use the registered information in the surrounding environment DB of the device (ST304).

[0106] As shown in FIG. 6(B), in the pedestrian terminal 1, when the ITS communication unit 15 receives an ITS communication message including DB usage information from the roadside unit 3 (Yes in ST111), in response to a transmission instruction from the processor 18, the wireless communication unit 16 transmits a wireless communication message to the roadside unit 3 requesting DB registration information (registration information of the surrounding environment DB) (ST112).

[0107] As shown in Figure 9 (B), in the roadside unit 3, when the wireless communication unit 32 receives a wireless communication message requesting DB registration information from the pedestrian terminal 1 (Yes in ST311), in response to a transmission instruction from the processor 34, the wireless communication unit 32 transmits a wireless communication message including the DB registration information (registration information of the surrounding environment DB) to the pedestrian terminal 1 (ST312).

[0108] At this time, all of the registered information in the surrounding environment DB of the roadside device 3 may be transmitted to the pedestrian terminal 1, or only a portion of the registered information that is likely to be used by the pedestrian terminal 1 may be transmitted to the pedestrian terminal 1. Specifically, the registered information within a predetermined range around the pedestrian terminal 1, particularly within a predetermined range located in the direction of travel of the pedestrian, may be transmitted to the pedestrian terminal 1.

[0109] As shown in Figure 6 (C), in the pedestrian terminal 1, when the wireless communication unit 16 receives a wireless communication message including DB registration information from the roadside unit 3 (Yes in ST121), the processor 18 registers the DB registration information (registration information of the surrounding environment DB) included in the received message in the surrounding environment DB of the device itself (ST122).

[0110] Next, as shown in FIG. 7, in the pedestrian terminal 1, the processor 18 acquires the position information of the pedestrian from the satellite positioning unit 11 (ST131).

[0111] Processor 18 also acquires state information of the pedestrian based on the detection result of state sensor 13 (ST132). Specifically, processor 18 measures the moving speed of the pedestrian based on the detection result of acceleration sensor 21. Processor 18 also measures the head direction (face direction) and moving direction of the pedestrian based on the detection results of acceleration sensor 21, gyro sensor 22, and geomagnetic sensor 23. Processor 18 also measures the gaze direction of the pedestrian based on the detection result of gaze sensor 24. Processor 18 also measures the gaze direction of the pedestrian based on the gaze direction and head direction of the pedestrian.

[0112] Next, the processor 18 acquires a real-time captured image from the camera 12 (ST133).

[0113] Next, in a matching target extraction process P15, the processor 18 extracts a candidate image corresponding to the gaze direction of the pedestrian from the surrounding environment DB of the own device as a matching target (ST134).

[0114] Next, the processor 18 performs image matching processing P16 by matching the candidate image extracted from the surrounding environment DB of the own device with the real-time captured image (ST135).

[0115] If this image matching is successful, i.e., if the candidate image and the captured image match (Yes in ST136), processor 18 acquires the location information associated with the successfully matched candidate image as the location information of the pedestrian's current location (ST137) as location information acquisition process P17.

[0116] (Second embodiment) Next, a second embodiment will be described. Note that points not particularly mentioned here are the same as those in the previous embodiment. Fig. 10 is an explanatory diagram showing an overview of the foot image matching process performed by the pedestrian terminal 1 according to the second embodiment. Fig. 11 is a block diagram showing the schematic configuration of the pedestrian terminal 1 according to the second embodiment.

[0117] In the first embodiment, visual positioning is performed using an image captured by camera 12 that captures an image in front of the pedestrian. On the other hand, in this modified example, visual positioning is performed using a front image that captures an image in front of the pedestrian and a foot image that captures an image of the pedestrian's feet.

[0118] In this modification, the pedestrian terminal 1 is equipped with a front camera 25 that captures an image in front of the pedestrian and a foot camera 26 that captures an image of the pedestrian's feet (see FIG. 11). Note that one camera may be configured to have a wide angle of view so that it can capture an image of a wide range including an image in front of and under the pedestrian's feet.

[0119] Road surfaces gradually deteriorate over time. For example, road surfaces are painted with white lines and other road markings using special traffic paint. These road markings can develop cracks and other deterioration. Asphalt pavement can also develop defects and other deterioration. The state of road surface deterioration has unique characteristics for each location. Therefore, the location where the underfoot image was captured can be identified based on the road surface characteristics.

[0120] Therefore, in this modification, a photographed image of the road surface at the registered point is registered in the surrounding environment DB in association with the position information of the registered point for use in foot verification. Also, as in the first embodiment, a photographed image taken in an imaging direction corresponding to the line of sight of the pedestrian looking ahead is registered in the surrounding environment DB in association with the position information of the registered point for use in forward verification.

[0121] Meanwhile, in the pedestrian terminal 1, the front camera 25 outputs a front image captured in front of the pedestrian in real time. As a visual positioning process P5, the processor 18 extracts candidate images for front matching (images captured at registered points) registered in the surrounding environment DB (forward matching target extraction process P21), and matches the candidate images for front matching with the real-time front image (forward image matching process P22, first matching process).

[0122] In addition, in the pedestrian terminal 1, the foot camera 26 outputs a foot image of the road surface under the pedestrian's feet in real time. As a visual positioning process P5, the processor 18 extracts candidate images for foot matching (images taken at registered points) registered in the surrounding environment DB (foot matching target extraction process P23), and matches the candidate images for foot matching with the real-time foot image (foot image matching process P24, second matching process).

[0123] In the forward image matching process P22, feature points are extracted from the forward image at the positions of objects such as buildings and road signs, while in the underfoot image matching process P24, feature points are extracted from the underfoot image at the positions of objects that appear on the road surface, such as white lines.

[0124] Next, processor 18 acquires position information of the pedestrian's current location based on the position information corresponding to the candidate image successfully matched in the forward image matching process P22 and the position information corresponding to the candidate image successfully matched in the feet image matching process P24 (position information acquisition process P17). At this time, the position information of the pedestrian's current location may be acquired by performing appropriate statistical processing (averaging) on ​​the position information based on the forward image matching process P22 and the position information based on the feet image matching process P24.

[0125] Furthermore, because the position information based on the foot image matching process P24 is more accurate than the position information based on the forward image matching process P22, the forward image matching process P22 may be used as a provisional positioning, and the foot image matching process P24 may be used as a final positioning, and the candidate images in the foot image matching process P24 may be narrowed down using the matching results of the forward image matching process P22. Specifically, the pedestrian's position based on the forward image matching process P22 is used as the pedestrian's provisional position, and candidate images for foot matching within a predetermined range around the provisional position are extracted from the surrounding environment DB, and the foot image matching process P24 is then performed. This reduces the processing load of image matching.

[0126] Here, the forward matching target extraction process P21 is the same as the matching target extraction process P15 (see FIG. 4) of the first embodiment. On the other hand, the feet matching target extraction process P23 is also substantially the same as the matching target extraction process P15 of the first embodiment, but in this case, a candidate image corresponding to a position on the road surface corresponding to the pedestrian's gaze direction is extracted.

[0127] In this embodiment, visual positioning is performed using both a forward image taken of the area in front of the pedestrian and a foot image taken of the pedestrian's feet, but visual positioning may also be performed using only a foot image.

[0128] (Third embodiment) Next, a third embodiment will be described. Note that points not particularly mentioned here are the same as those in the above-described embodiments. Fig. 12 is an explanatory diagram showing an overview of control performed by the pedestrian terminal 1 according to the third embodiment.

[0129] Road features, i.e., distinctive objects visible to pedestrians on the road, such as buildings and road signs, serve as map indicators. Therefore, in visual positioning, by focusing on the features included in the images captured by the camera 12, it is possible to improve the efficiency of the matching process and the accuracy of positioning.

[0130] Therefore, in this embodiment, the pedestrian terminal 1 selects a feature of interest from among the features present around the pedestrian, obtains the direction in which the feature of interest is located relative to the pedestrian, and in visual positioning, extracts candidate images including the feature of interest from the surrounding environment information based on the direction in which the feature of interest is located as the optimal matching direction.

[0131] Furthermore, in this embodiment, a camera 12 with a variable shooting direction is employed in the pedestrian terminal 1. Based on the direction in which the noted feature exists, the pedestrian terminal 1 determines whether or not the noted feature is included in the shooting range of the camera 12 (detection range of the external sensor), and if the noted feature is not included in the shooting range of the camera 12, controls the shooting direction of the camera 12 (detection direction of the external sensor) so that the noted feature is included in the shooting range of the camera 12.

[0132] Furthermore, in this embodiment, information (e.g., position information) about feature objects is registered in advance in a feature DB (feature database). The pedestrian terminal 1 can acquire the direction in which a feature of interest exists relative to the pedestrian, based on the position information of the feature objects registered in the feature DB and the position information of the pedestrian acquired by satellite positioning (standard positioning). Note that the feature DB may not only store two-dimensional position information (latitude and longitude) of feature objects, but also three-dimensional position information including information about the height of the feature objects.

[0133] Furthermore, at night, buildings and road signs become less noticeable, reducing the efficiency of the matching process. Therefore, in this embodiment, information about lighting devices (including illumination devices) such as streetlights and traffic lights installed along roads is registered as feature objects in the feature DB. When selecting a feature of interest, lighting devices are given priority in selection at night. This allows for the efficiency of the matching process in visual positioning to be improved, even at night.

[0134] The registration information (feature object information) of the feature object DB is distributed from the roadside unit 3 to the pedestrian terminal 1, similar to the surrounding environment DB. The registration information of the feature object DB may be managed by a cloud computer and distributed from the cloud computer to the pedestrian terminal 1. In this case, the information may be distributed from the cloud computer to the pedestrian terminal 1 via the roadside unit 3, or may be distributed from the cloud computer to the pedestrian terminal 1 using high-speed cellular communication such as 5G without passing through the roadside unit 3.

[0135] Next, a schematic configuration of the pedestrian terminal 1 according to the third embodiment will be described. Fig. 13 is a block diagram showing a schematic configuration of the pedestrian terminal 1.

[0136] In this embodiment, the pedestrian terminal 1 includes a camera driving unit 19. Furthermore, a memory 17 of the pedestrian terminal 1 stores registration information of the feature DB. Furthermore, a processor 18 of the pedestrian terminal 1 performs a feature selection process P31, a feature direction acquisition process P32, a feature photograph determination process P33, and a camera control process P34. The rest is substantially the same as in the first embodiment (see FIG. 4).

[0137] The camera driving unit 19 changes the shooting direction of the camera 12 based on instructions from the processor 18.

[0138] In the feature selection process P31, the processor 18 selects a feature of interest from among the features present around the pedestrian based on the pedestrian's position information and the registered information in the feature DB of the device itself. At this time, the feature of interest is selected based on the visibility of the feature. For example, a feature that is located close to the pedestrian, appears large to the pedestrian, and stands out due to its color or shape, etc., and can be easily recognized by the pedestrian, is selected as the feature of interest.

[0139] In the feature direction acquisition process P32, the processor 18 acquires the feature direction, that is, the direction in which the noted feature exists relative to the pedestrian, based on the position information of the pedestrian and the position information of the noted feature.

[0140] In the feature capture determination process P33, processor 18 determines whether or not the feature of interest is included in the capture range of camera 12, i.e., whether or not the feature of interest appears in the image captured by camera 12, based on the feature direction, i.e., the direction in which the feature of interest exists relative to the pedestrian, and the current capture range of camera 12. Here, the current capture range of camera 12 is determined based on the current capture direction and angle of view of camera 12. Furthermore, the capture direction of camera 12 is determined based on the angle of change from the initial position of camera 12 and the head direction (face orientation) of the pedestrian.

[0141] In the camera control process P34, if the target feature is not included in the current shooting range of the camera 12, the processor 18 controls the camera driving unit 19 to adjust the shooting direction of the camera 12 so that the target feature is included in the shooting range of the camera 12.

[0142] On the other hand, in the matching target extraction process P15 of the visual positioning process P5, the processor 18 extracts candidate images containing the target feature from the surrounding environment DB of the device as a matching target based on the feature direction, i.e., the direction in which the target feature is located relative to the pedestrian.

[0143] Next, a schematic configuration of the roadside unit 3 according to the third embodiment will be described. FIG.

[0144] In this embodiment, the memory 33 of the roadside device 3 stores the registration information of the feature DB. Also, the processor 34 of the roadside device 3 performs the feature DB management process. The rest is substantially the same as the first embodiment (see FIG. 5).

[0145] In the feature DB management process, the processor 34 manages the feature DB. Specifically, in response to a request from the pedestrian terminal 1, the processor 34 distributes registration information of the feature DB from the wireless communication unit 32 to the pedestrian terminal 1.

[0146] Next, the operation procedure of the pedestrian terminal 1 according to the third embodiment will be described. Fig. 15 is a flow diagram showing the operation procedure of the pedestrian terminal 1. Note that the pedestrian terminal 1 performs the same processing as in the first embodiment (see Figs. 6(A), (B), and (C)). Also, the operation procedure of the roadside device 3 is the same as in the first embodiment (see Fig. 9).

[0147] In the pedestrian terminal 1, as in the first embodiment (see FIG. 7), the processor 18 acquires the position information of the pedestrian from the satellite positioning unit 11 (ST131), and also acquires the state information of the pedestrian (movement speed, head direction (face direction), traveling direction, gaze direction, and gaze direction) based on the detection results of the state sensor 13 (ST132).

[0148] Next, in a feature selection process P31, processor 18 selects a feature of interest from among the features present around the pedestrian based on the position information of the pedestrian and the registered information in the feature DB of the own device (ST141).

[0149] Next, as a feature direction acquisition process P32, processor 18 acquires the feature direction, i.e., the direction in which the feature of interest is located relative to the pedestrian, based on the position information of the pedestrian and the position information of the feature of interest (ST142).

[0150] Next, as a feature photographing determination process P33, processor 18 determines whether the feature of interest is included in the photographing range of camera 12, i.e., whether the feature of interest is captured in the image photographed by camera 12, based on the feature direction, i.e., the direction in which the feature of interest is located relative to the pedestrian, and the current photographing range of camera 12 (ST143).

[0151] Here, if the feature of interest is not included in the shooting range of the camera 12 (No in ST143), the processor 18 controls the camera driving unit 19 to adjust the shooting direction of the camera 12 so that the feature of interest is included in the shooting range of the camera 12 as camera control processing P34 (ST144).

[0152] Next, the processor 18 acquires a real-time captured image from the camera 12 (ST133).

[0153] Next, in a matching target extraction process P15, the processor 18 extracts candidate images including a notable feature from the surrounding environment DB of the own device as matching targets (ST145).

[0154] The subsequent processes (ST135 to ST137) are the same as those in the first embodiment (see FIG. 7).

[0155] As shown in FIG. 9(A), the roadside unit 3 transmits an ITS communication message including DB usage information to the pedestrian terminal 1 (ST304). In this embodiment, this ITS communication message includes DB usage information indicating that the registration information in the surrounding environment DB and the registration information in the feature DB can be used. As shown in FIG. 9(B), the roadside unit 3 transmits a wireless communication message including DB registration information to the pedestrian terminal 1 (ST312). In this embodiment, this wireless communication message includes the registration information in the surrounding environment DB and the registration information in the feature DB as the DB registration information. As a result, as shown in FIG. 6(C), the pedestrian terminal 1 receives the wireless communication message from the roadside unit 3 (Yes in ST121) to acquire the registration information in the surrounding environment DB and the feature DB as the DB registration information, and then registers the registration information in the surrounding environment DB and the feature DB in the pedestrian terminal 1's surrounding environment DB and the feature DB, respectively (ST122).

[0156] (Fourth embodiment) Next, a fourth embodiment will be described. Note that points not particularly mentioned here are the same as those in the above-described embodiments. Fig. 16 is an explanatory diagram showing an overview of control performed by the pedestrian terminal 1 according to the fourth embodiment.

[0157] In the third embodiment, a camera 12 with a variable shooting direction is adopted in the pedestrian terminal 1, and when the target feature is not included in the shooting range of the camera 12, i.e., when the target feature is not captured in the current image captured by the camera 12, the pedestrian terminal 1 controls the shooting direction of the camera 12 so that the target feature is included in the shooting range of the camera 12.

[0158] On the other hand, in this embodiment, the pedestrian terminal 1 employs a fixed camera 12 that captures an image directly in front of the pedestrian, and when the target feature is not included in the imaging range of the camera 12, the pedestrian terminal 1 performs guidance control to prompt the pedestrian to change the imaging direction of the camera 12 so that the target feature is included in the imaging range of the camera 12. Particularly in this embodiment, as the guidance control, a guidance screen is displayed on the AR display 14 to prompt the pedestrian to turn their face left or right.

[0159] In this embodiment, the pedestrian terminal 1 is configured as a wearable device attached to the head of the pedestrian, and the camera 12 is provided so as to face the front of the pedestrian, so that the direction of the pedestrian's face matches the shooting direction of the camera 12. Therefore, by prompting the pedestrian to change the direction of their face left or right on the guidance screen, the shooting direction (shooting range) of the camera 12 can be changed left or right.

[0160] For example, if the feature of interest is located to the right of the current shooting range, the pedestrian is guided to look to the right, and if the feature of interest is located to the left of the current shooting range, the pedestrian is guided to look to the left. In the example shown in Fig. 16, since the feature of interest is located to the left of the current shooting range, an image of an arrow guiding the pedestrian to look to the left is displayed on the screen of the AR display 14.

[0161] A navigation screen that displays the current position of the pedestrian on a map is displayed on the AR display 14 of the pedestrian terminal 1. Information from the feature DB may be displayed on this navigation screen. For example, the position of the feature may be displayed on the map. Furthermore, an image showing the direction of travel of the pedestrian or an image showing the positional relationship between the pedestrian and the feature may be displayed on the map.

[0162] Next, a schematic configuration of the pedestrian terminal 1 according to the fourth embodiment will be described. Fig. 17 is a block diagram showing a schematic configuration of the pedestrian terminal 1.

[0163] In this embodiment, the processor 18 of the pedestrian terminal 1 performs a feature object selection process P31, a feature object direction acquisition process P32, a feature object photographing determination process P33, and a pedestrian guidance process P41. The rest is substantially the same as in the first embodiment (see FIG. 4). The feature object selection process P31, the feature object direction acquisition process P32, and the feature object photographing determination process P33 are the same as in the third embodiment.

[0164] In the pedestrian guidance process P41, when the target feature is not included in the imaging range of the camera 12, the processor 18 performs guidance control to prompt the pedestrian to change the imaging direction of the camera 12 so that the target feature is included in the imaging range of the camera 12. Specifically, as the guidance control, the processor 18 turns the face of the pedestrian in a specified direction, that is, displays on the AR display 14 a screen that prompts the pedestrian wearing the pedestrian terminal 1 to change the direction of his or her head (face direction).

[0165] In this embodiment, as guidance control, a screen guiding the pedestrian to change the direction of their face is displayed on the AR display 14, but audio guiding the pedestrian to change the direction of their face may also be output from a speaker (not shown).

[0166] Next, the operation procedure of the pedestrian terminal 1 according to the fourth embodiment will be described. Fig. 18 is a flow diagram showing the operation procedure of the pedestrian terminal 1. Note that the pedestrian terminal 1 performs the same processing as in the first embodiment (see Figs. 6(A), (B), and (C)). Also, the operation procedure of the roadside device 3 is the same as in the first embodiment (see Fig. 9).

[0167] In the pedestrian terminal 1, as in the third embodiment (see FIG. 15), the processor 18 performs processing from acquiring the position information of the pedestrian by the satellite positioning unit 11 to determining whether or not the target feature is included in the current shooting range of the camera 12 (ST131, ST132, ST141 to ST143).

[0168] If the target feature is not included in the current shooting range of the camera 12, a screen that guides the pedestrian to change the direction of their head (direction of their face) is displayed on the AR display 14 (ST151).

[0169] The subsequent processes (ST133, ST145, ST135 to ST137) are the same as those in the third embodiment.

[0170] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0171] The pedestrian device, mobile device, positioning system, and positioning method according to the present invention have the effect of improving the efficiency of the matching process in visual positioning in order to shorten the processing time of visual positioning, and are useful as pedestrian devices that are held by pedestrians to perform positioning to measure the current position of the pedestrian, mobile devices, positioning systems, and positioning methods that are held by moving bodies such as pedestrians and vehicles to perform positioning to measure the current position of the moving body, etc. [Explanation of symbols]

[0172] 1 Pedestrian terminal (pedestrian device, mobile device, computer) 2. In-vehicle terminal 3 Roadside unit 11 Satellite positioning unit 12 Camera 13 Status Sensor 14 AR Display 17. Memory 18 processors 19 Camera drive unit 21 Acceleration sensor 22 Gyro sensor 23 Geomagnetic sensor 24 Eye Sensor 25 Front camera 26 Foot Camera

Claims

1. an external sensor that detects objects around the pedestrian; a memory that stores surrounding environment information relating to the surrounding environment of a pedestrian; a processor that performs processing related to visual positioning that estimates a current position of a pedestrian by comparing the detected image of the external sensor with the surrounding environment information, The processor: Inferring the optimal matching direction that is likely to result in successful matching; In the visual positioning, extracting candidate information corresponding to the optimum matching direction from the surrounding environment information as a matching target; comparing the detected image of the external sensor with the candidate information; The pedestrian device acquires location information associated with the candidate information for which matching has been successful as current location information of the pedestrian.

2. A state sensor is provided to detect the state of a pedestrian, The processor: Measure the gaze direction of the pedestrian based on the detection result of the state sensor; 2. The pedestrian device according to claim 1, wherein the candidate information corresponding to the gaze direction as the optimum verification direction is extracted from the surrounding environment information as a verification target.

3. It consists of a wearable device attached to the pedestrian's body, the state sensor detects at least one of a head direction and a line of sight direction of a pedestrian; The processor:

3. The pedestrian device according to claim 2, wherein the gaze direction is measured based on at least one of the head direction and the line of sight direction.

4. The processor: The pedestrian device according to claim 2, wherein the candidate information including the feature point located in the gaze direction is extracted from the surrounding environment information as a matching target.

5. the external sensor is a camera that captures an image of the pedestrian's surroundings, The processor: In the visual positioning, 2. The pedestrian device according to claim 1, wherein an image of the pedestrian's surroundings photographed by the camera is collated with the candidate information.

6. the external sensor is a camera that captures an image of the pedestrian's feet, The processor:

2. The pedestrian device according to claim 1, wherein an image of the pedestrian's feet photographed by the camera is collated with the candidate information.

7. The memory includes: storing feature information including location information of features that serve as indicators on a map; The processor: Based on the position information of the pedestrian acquired by standard positioning and the feature information, a feature of interest is selected from feature objects present around the pedestrian, and a direction in which the feature of interest exists relative to the pedestrian is acquired; In the visual positioning, 2. The pedestrian device according to claim 1, wherein the candidate information including the target feature is extracted from the surrounding environment information based on the direction in which the target feature exists as the optimum matching direction.

8. The processor: Based on the direction in which the feature of interest exists, determining whether the target feature is included in the detection range of the external sensor; If the target feature is not included in the detection range of the external sensor, 8. The pedestrian device according to claim 7, wherein the detection direction of the external sensor is controlled so that the target feature is included in the detection range of the external sensor.

9. The processor: Based on the direction in which the feature of interest exists, determining whether the target feature is included in the detection range of the external sensor; If the target feature is not included in the detection range of the external sensor, 8. The pedestrian device according to claim 7, wherein guidance control is performed to prompt the pedestrian to take action to change the detection direction of the external sensor so that the target feature is included in the detection range of the external sensor.

10. The feature information is 8. The pedestrian device according to claim 7, wherein the features include information about lighting devices installed along the road.

11. an external sensor that detects objects around the moving object; a memory that stores surrounding environment information relating to the surrounding environment of the moving object; a processor that performs processing related to visual positioning that estimates a current position of a moving object by comparing the detected image of the external sensor with the surrounding environment information, The processor: Inferring the optimal matching direction that is likely to result in successful matching; In the visual positioning, extracting candidate information corresponding to the optimum matching direction from the surrounding environment information as a matching target; comparing the detected image of the external sensor with the candidate information; The mobile device acquires location information associated with the candidate information that has been successfully matched as current location information of the mobile device.

12. A positioning system configured with one or more computers that executes a process for acquiring position information of a mobile body in a mobile body device, an external sensor provided in the mobile device and configured to detect objects around the mobile device; The computer a memory that stores surrounding environment information relating to the surrounding environment of the moving object; a processor that performs processing related to visual positioning that estimates a current position of a moving object by comparing the detected image of the external sensor with the surrounding environment information, The processor: Inferring the optimal matching direction that is likely to result in successful matching; In the visual positioning, extracting candidate information corresponding to the optimum matching direction from the surrounding environment information as a matching target; comparing the detected image of the external sensor with the candidate information; A positioning system characterized in that location information associated with the candidate information that has been successfully matched is acquired as current location information of the mobile object.

13. A positioning method in which one or more computers execute a process for acquiring position information of a mobile body in a mobile body device, The computer performing a visual positioning process for estimating the current position of the mobile body by comparing an image detected by an external sensor that detects objects around the mobile body with environmental information about the mobile body's surrounding environment; Inferring the optimal matching direction that is likely to result in successful matching; In the visual positioning, extracting candidate information corresponding to the optimum matching direction from the surrounding environment information as a matching target; comparing the detected image of the external sensor with the candidate information; A positioning method characterized in that location information associated with the candidate information that has been successfully matched is acquired as current location information of the mobile object.

Images