Pedestrian device and method for measuring pedestrian position

Abstract

To reduce a processing load when positioning a pedestrian by using a camera image obtained by imaging a road surface on which the pedestrian walks.SOLUTION: A pedestrian terminal 1 comprises: a camera 11 which generates a step camera image and a pedestrian periphery camera image; a memory 17 which stores road surface registration information in which position information is associated with a road surface camera image of a registration point and fixed object registration information in which position information is associated with feature information of a fixed object in a periphery camera image obtained by imaging the periphery; and a processor 18. The processor executes temporary positioning of a pedestrian by using position information in the fixed object registration information by comparison between feature information of an object extracted from the pedestrian periphery camera image with feature information of the fixed object, collates each collation candidate image extracted from the road surface camera image with the step camera image on the basis of the result of the temporary positioning, and acquires position information of the registration point associated with the collation candidate image succeeded in collation as position information of a current place of the pedestrian.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to a pedestrian device that is carried by a pedestrian and performs positioning to obtain position information of the pedestrian, and to a positioning method thereof. [Background technology]

[0002] In a safe driving support wireless system that uses ITS (Intelligent Transport System), vehicle position information is exchanged between in-vehicle terminals to prevent accidents between vehicles, and vehicle and pedestrian position information is exchanged between in-vehicle terminals and pedestrian terminals to prevent accidents between vehicles and pedestrians.

[0003] In-vehicle terminals and pedestrian terminals mainly obtain the position information of vehicles and pedestrians by satellite positioning, but various positioning methods can be adopted, such as positioning using PDR (Pedestrian Dead Reckoning). In this case, a positioning method that can obtain highly accurate position information is desired to prevent traffic accidents.

[0004] Therefore, it is conceivable to photograph the surroundings of the vehicle or pedestrian with a camera and measure the positions of the vehicle or pedestrian based on the camera image. As a positioning method using such camera images, a technology is known that detects white lines painted on the road surface based on a camera image of the road and recognizes the driving lane in which the vehicle is traveling as the vehicle's position information (see Patent Documents 1-3). In addition, a technology is also known that measures the position of the device itself by focusing on landmark objects (such as buildings around the road) that appear in a camera image of the front. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 2754871 [Patent Document 2] Patent No. 3333223 [Patent Document 3] Japanese Patent Application Publication No. 6-149360 Summary of the Invention [Problem to be solved by the invention]

[0006] Since pedestrians change their moving speed and direction significantly compared to vehicles, etc., there are cases where highly accurate positioning cannot be achieved when the positioning method according to the conventional technology described above is adopted as is to obtain position information of a pedestrian's current location. In addition, when positioning a pedestrian using a camera image capturing the road surface, it is desirable to reduce the processing load.

[0007] In view of the above background, a primary objective of the present disclosure is to provide a pedestrian device and a positioning method thereof that can reduce the processing load when positioning pedestrians using camera images capturing a road surface on which pedestrians walk. [Means for solving the problem]

[0008] The pedestrian device of the present disclosure includes a foot camera that sequentially generates foot camera images by capturing an image of the road surface under the feet of the pedestrian, a surrounding camera that sequentially generates pedestrian surrounding camera images by capturing an image of at least one of the front, rear, left and right directions of the pedestrian, a memory that stores road surface registration information in which a plurality of road surface camera images capturing the road surface at a registration point are associated with position information of the registration point, and fixed object registration information in which position information is associated with feature information on fixed objects included in surrounding camera images capturing the surroundings, and a processor that executes a process of acquiring position information of the current location of the pedestrian, and the processor ... The sensor is configured to extract feature information on objects included in the pedestrian surrounding camera image, compare the feature information on the object with feature information on the fixed object in the fixed object registration information, and perform a provisional positioning of the pedestrian using the position information in the fixed object registration information, extract some of the multiple road camera images stored in the memory as match candidate images based on the result of the provisional positioning, match the foot camera image with each of the match candidate images, and obtain position information of the registered point associated with the match candidate image that was successfully matched as position information of the pedestrian's current location.

[0009] The positioning method of a pedestrian device disclosed herein is a positioning method of a pedestrian device for acquiring current position information of a pedestrian, which sequentially generates foot camera images by photographing the road surface under the pedestrian's feet with a camera, and sequentially generates pedestrian surroundings camera images by photographing at least one of the front, back, left, and right directions of the pedestrian with a camera, and stores in a memory road surface registration information in which position information of the registered points is associated with a plurality of road surface camera images photographing the road surface at the registered points, and fixed object registration information in which position information is associated with feature information on fixed objects included in surrounding camera images photographing the surroundings, The system is configured to extract feature information related to objects included in the pedestrian surrounding camera image, compare the feature information related to the object with feature information related to the fixed object in the fixed object registration information, and perform a provisional positioning of the pedestrian using the position information in the fixed object registration information. Based on the result of the provisional positioning, some of the multiple road camera images stored in the memory are extracted as match candidate images, match the foot camera image with each of the match candidate images, and obtain the position information of the registered point associated with the match candidate image that was successfully matched as the position information of the pedestrian's current location. Effect of the Invention

[0010] According to the present disclosure, when pedestrians are located using camera images capturing a road surface on which pedestrians walk, the processing load of image matching can be reduced. [Brief description of the drawings]

[0011] [Figure 1] Overall configuration diagram of the traffic safety support system according to the first embodiment [Diagram 2] FIG. 1 is an explanatory diagram showing an overview of an image matching process performed by a pedestrian terminal according to a first embodiment; [Diagram 3] FIG. 1 is an explanatory diagram showing a setting status of a registered point according to a first embodiment; [Figure 4] FIG. 1 is an explanatory diagram showing the registered contents of an image position DB according to the first embodiment; [Diagram 5] FIG. 1 is an explanatory diagram showing an overview of a provisional positioning process performed by a pedestrian terminal according to a first embodiment; [Figure 6] FIG. 1 is a block diagram showing a schematic configuration of a pedestrian terminal and a roadside unit according to a first embodiment; [Figure 7] FIG. 1 is a flow diagram showing an operation procedure of the pedestrian terminal 1 according to the first embodiment. [Figure 8] FIG. 1 is a flow diagram showing an operation procedure of the pedestrian terminal 1 according to the first embodiment. [Figure 9] FIG. 1 is a flow diagram showing an operation procedure of the in-vehicle terminal 2 according to the first embodiment. [Figure 10] FIG. 1 is a flow diagram showing an operation procedure of a roadside device 3 according to a first embodiment. [Figure 11] FIG. 11 is a block diagram showing a schematic configuration of a pedestrian terminal according to a second embodiment. [Figure 12] FIG. 11 is a flow diagram showing an operation procedure of the pedestrian terminal according to the second embodiment. [Figure 13] FIG. 13 is a block diagram showing a schematic configuration of a pedestrian terminal according to a third embodiment. [Figure 14] FIG. 11 is a flow chart showing an operation procedure of the pedestrian terminal according to the third embodiment. [Figure 15] FIG. 13 is a flowchart showing an operation procedure of a pedestrian terminal according to a first modified example of the third embodiment. [Figure 16] FIG. 13 is a flowchart showing an operation procedure of a pedestrian terminal according to a second modified example of the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The first invention made to solve the above problems includes a foot camera that sequentially generates foot camera images by photographing the road surface under the feet of a pedestrian, a surrounding camera that sequentially generates pedestrian surrounding camera images by photographing at least one of the front, back, left and right directions of the pedestrian, a memory that stores road surface registration information in which a plurality of road surface camera images taken of the road surface at a registered point correspond to position information of the registered point, and fixed object registration information in which position information of a fixed object included in a surrounding camera image taken of the surroundings correspond to feature information of the fixed object included in the surrounding camera image, and a processor that executes a process to obtain position information of the current location of the pedestrian. The processor extracts feature information on objects included in the pedestrian surrounding camera image, compares the feature information on the object with feature information on the fixed object in the fixed object registration information, and performs a provisional positioning of the pedestrian using the position information in the fixed object registration information. Based on the result of the provisional positioning, it extracts some of the multiple road camera images stored in the memory as match candidate images, matches the foot camera image with each of the match candidate images, and obtains the position information of the registered point corresponding to the match candidate image that was successfully matched as the position information of the pedestrian's current location.

[0013] According to this system, a match candidate image extracted from road camera images based on the results of provisional positioning is compared with a foot camera image, and the location information of the registered point corresponding to the successfully matched match candidate image is obtained as the location information of the pedestrian's current location. This reduces the processing load when pedestrians are positioned using camera images (foot camera images and road camera images) that capture the road surface on which the pedestrian is walking.

[0014] In addition, the second invention is configured to further include a receiver that receives a satellite positioning signal, and the processor acquires the fixed object registration information from another device in accordance with location information obtained from the satellite positioning signal, and stores the acquired fixed object registration information in the memory.

[0015] This makes it possible to obtain only the required range of fixed object registration information according to the position information obtained from the satellite positioning signal.

[0016] In a third aspect of the present invention, the foot camera and the surrounding camera are configured as a single 360-degree camera.

[0017] This makes it possible to obtain foot camera images and pedestrian surroundings camera images as needed without complicating the configuration of the pedestrian device.

[0018] In addition, a fourth invention is configured such that the processor is capable of interrupting the provisional positioning, and while the provisional positioning is interrupted, sequentially calculates the amount of movement of the pedestrian, and based on the calculation result of the amount of movement, extracts a portion of the multiple road camera images stored in the memory as the match candidate image.

[0019] According to this, the processing load on the processor can be reduced by interrupting provisional positioning, which involves a relatively large processing load, and extracting match candidate images by calculating the amount of movement, which involves a relatively small processing load.

[0020] In addition, a fifth invention is configured such that the processor acquires the amount of movement of the pedestrian by at least one of a self-position estimation process based on pedestrian autonomous navigation and a self-position estimation process based on the pedestrian surrounding camera image generated by the surrounding camera.

[0021] This makes it possible to obtain the amount of pedestrian movement through simple processing.

[0022] In addition, a sixth invention is configured such that the processor repeatedly performs the provisional positioning of the pedestrian, sequentially stores the results of the provisional positioning in the memory, determines whether or not an obstruction has occurred in the shooting area of ​​the surrounding camera based on the pedestrian surrounding camera images, and if it determines that an obstruction has occurred in the shooting area of ​​the surrounding camera, obtains the past provisional positioning results stored in the memory as the latest provisional positioning results.

[0023] This makes it possible to avoid the use of inappropriate surrounding camera images of a pedestrian when an obstruction occurs in the shooting area of ​​the surrounding camera, and to reliably obtain location information of the pedestrian's current location.

[0024] In addition, a seventh invention is configured in such a way that the processor is capable of interrupting the provisional positioning, and while the provisional positioning is interrupted, sequentially calculates the amount of movement of the pedestrian, and based on the calculation result of the amount of movement, extracts a portion of the multiple road camera images stored in the memory as the match candidate image, and when it is determined that the occlusion has occurred in the shooting area of ​​the surrounding camera, calculates the duration of the occlusion, and when the duration of the occlusion becomes equal to or longer than a threshold, interrupts the provisional positioning, and obtains the amount of movement of the pedestrian by at least one of a self-position estimation process based on pedestrian autonomous navigation and a self-position estimation process based on the pedestrian surrounding camera images generated by the surrounding camera.

[0025] With this, when the duration of occlusion occurring in the shooting area of ​​the surrounding cameras becomes long, the current location information can be appropriately obtained based on the amount of movement of the pedestrian.

[0026] In addition, the eighth invention is configured to further include a communication unit that performs wireless communication with at least one of an in-vehicle device and a roadside device mounted on the vehicle, and when the processor determines that an obstruction has occurred in the shooting area of ​​the surrounding camera, it transmits a message regarding the occurrence of the obstruction to at least one of the in-vehicle device and the roadside device that perform the wireless communication via the communication unit.

[0027] This makes it possible to notify the vehicle-mounted device of a situation in which the surroundings of the pedestrian (for example, the front) are obstructed, thereby improving the safety of the pedestrian and the vehicle.

[0028] A ninth aspect of the present invention is a positioning method for a pedestrian device for acquiring current position information of a pedestrian, which further comprises the steps of: photographing a road surface under the feet of the pedestrian with a camera, thereby sequentially generating foot camera images; photographing at least one of the front, rear, left and right directions of the pedestrian with a camera, thereby sequentially generating pedestrian surroundings camera images; storing in a memory road surface registration information in which position information of the registered points is associated with a plurality of road surface camera images photographing the road surface at the registered points; and fixed object registration information in which position information is associated with feature information on fixed objects included in peripheral camera images photographing the surroundings; The system is configured to extract feature information relating to objects included in the pedestrian-surrounding camera image, compare the feature information relating to the objects with feature information relating to the fixed objects in the fixed object registration information, and perform a provisional positioning of the pedestrian using the position information in the fixed object registration information. Based on the results of the provisional positioning, some of the multiple road camera images stored in the memory are extracted as match candidate images, match the feet camera image with each of the match candidate images, and obtain the position information of the registered point associated with the match candidate image that was successfully matched as the position information of the pedestrian's current location.

[0029] According to this system, a match candidate image extracted from road camera images based on the results of provisional positioning is compared with a foot camera image, and the location information of the registered point corresponding to the successfully matched match candidate image is obtained as the location information of the pedestrian's current location. This reduces the processing load when pedestrians are positioned using camera images (foot camera images and road camera images) that capture the road surface on which the pedestrian is walking.

[0030] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0031] (First embodiment) FIG. 1 is an overall configuration diagram of a traffic safety support system according to the first embodiment.

[0032] The traffic safety support system supports the traffic safety of a pedestrian W and a vehicle V, and includes a pedestrian terminal 1 (pedestrian device), an in-vehicle terminal 2 (in-vehicle device), and a road-side unit 3.

[0033] ITS communication is performed between the pedestrian terminal 1, the vehicle-mounted terminal 2, and the roadside unit 3. This ITS communication is wireless communication using a frequency band (e.g., 700 MHz band or 5.8 GHz band) adopted in a safe driving support wireless system using an ITS (Intelligent Transport System). In this embodiment, the ITS communication between the pedestrian terminal 1 and the vehicle-mounted terminal 2 is appropriately referred to as pedestrian-vehicle communication, the ITS communication between the pedestrian terminal 1 and the roadside unit 3 is appropriately referred to as road-to-pedestrian communication, and the ITS communication between the vehicle-mounted terminal 2 and the roadside unit 3 is appropriately referred to as road-to-vehicle communication. In addition, ITS communication is also performed between the vehicle-mounted terminals 2, and this ITS communication is appropriately referred to as vehicle-to-vehicle communication.

[0034] The pedestrian terminal 1 is carried by its user, a pedestrian W. This pedestrian terminal 1 transmits and receives messages including location information and the like to and from an in-vehicle terminal 2 by ITS communication (pedestrian-to-vehicle communication). In this way, the pedestrian terminal 1 determines the risk of collision between the pedestrian W and a vehicle, and performs an action to alert the pedestrian W if there is a risk of collision.

[0035] The in-vehicle terminal 2 is mounted on the vehicle V. This in-vehicle terminal 2 transmits and receives messages including location information and the like to and from the pedestrian terminal 1 by ITS communication (pedestrian-to-vehicle communication). In this way, the in-vehicle terminal 2 determines the risk of collision between the pedestrian W and the vehicle V, and performs an attention-calling operation for the driver of the vehicle V if there is a risk of collision. Note that the attention-calling operation by the in-vehicle terminal 2 may be performed using, for example, a car navigation device.

[0036] The roadside unit 3 is installed at a road intersection or the like. This roadside unit 3 distributes various information such as traffic information to the pedestrian terminal 1 and the in-vehicle terminal 2 by ITS communication (road-to-pedestrian communication, road-to-vehicle communication). In addition, the roadside unit 3 notifies the in-vehicle terminal 2 and the pedestrian terminal 1 of the presence of a vehicle V or a pedestrian W located in the vicinity of the roadside unit 3 by ITS communication (road-to-vehicle communication, road-to-pedestrian communication). This allows the vehicle V and the pedestrian W to prevent collisions at intersections that are out of line of sight.

[0037] The pedestrian terminal 1 is equipped with a camera 11 (a foot camera, a surrounding camera). The camera 11 can generate a camera image of the road surface under the feet of the pedestrian W (hereinafter referred to as a foot camera image) and a camera image of the front (moving direction) of the pedestrian W (hereinafter referred to as a front camera image). The foot camera image and the front camera image can be generated at a predetermined frame rate as frame images constituting a video (video). In this embodiment, the camera 11 is provided integrally with the main body of the pedestrian terminal 1. However, the camera 11 may be provided separately from the main body of the pedestrian terminal 1 in consideration of convenience of shooting (shooting direction, angle of view, etc.). In that case, the camera 11 is connected to the main body of the pedestrian terminal 1 so as to be able to communicate with each other by wire or wirelessly.

[0038] The camera 11 may be composed of multiple cameras (see the foot camera 11A and the front camera 11B in FIG. 6) that generate foot camera images and front camera images, respectively. Alternatively, the camera 11 may be composed of a single camera (e.g., a 360-degree camera) that can capture a wide range so that the foot camera images and the front camera images (or images equivalent thereto) can be generated. By including a single camera that can capture a wide range (i.e., that can cover a required capture area), the pedestrian terminal 1 can obtain foot camera images and front camera images as necessary without complicating its configuration.

[0039] In this embodiment, for convenience of explanation, an example is shown in which the camera 11 generates a foot camera image capturing the road surface under the feet of the pedestrian W and a front camera image capturing the front of the pedestrian W, but instead of the front camera image, a camera image capturing another direction may be generated. That is, the front camera 11B (surrounding camera) may be a camera capable of generating an image capturing at least one of the front, back, left and right directions of the pedestrian W (pedestrian surrounding camera image). For example, for privacy protection, the camera 11 may capture the rear of the pedestrian W (the direction opposite to the moving direction) instead of capturing the front of the pedestrian W. Alternatively, the camera 11 may be configured by one camera (e.g., a 360-degree camera) capable of capturing a wide range so that a camera image (or an image equivalent thereto) in at least one of the front, back, left and right directions of the pedestrian W can be generated in addition to the foot camera image. Note that the orientation (capturing direction) of the camera 11 does not need to strictly match any of the front, back, left and right directions of the pedestrian W.

[0040] In the example shown in FIG. 1, the pedestrian terminal 1 is a glasses-type wearable device (smart glasses) that is attached to the head of the pedestrian W. The pedestrian terminal 1 has an AR display and is provided with a function of realizing AR (Augmented Reality) on the AR display. AR is realized by superimposing a virtual object on the AR display on a real space that is within the real field of view of the pedestrian W. On the AR display, an image representing a risk of collision with a vehicle, an image of a vehicle that is not directly visible to the pedestrian W at an intersection with no line of sight, and the like are displayed as virtual objects. Note that the pedestrian terminal 1 may be configured by a plurality of devices that can communicate with each other. For example, the pedestrian terminal 1 may be configured by a head-mounted device attached to the head of the pedestrian W and a main body device that is provided separately from the head-mounted device and is placed outside the head of the pedestrian W.

[0041] Next, an overview of image matching positioning performed by the pedestrian terminal 1 according to the first embodiment will be described. FIG. 2 is an explanatory diagram showing an overview of the image matching process performed by the pedestrian terminal 1. FIG. 3 is an explanatory diagram showing a setting status of a registered point. FIG. 4 is an explanatory diagram showing the registered contents of the image position DB. FIG. 5 is an explanatory diagram showing an overview of tentative positioning performed by the pedestrian terminal 1. Although a pedestrian terminal has been described in this embodiment, a similar mechanism can also be applied to an in-vehicle terminal.

[0042] Road surfaces gradually deteriorate over time. For example, road surfaces are painted with white lines and other road markings using a special paint (traffic paint). These road markings can deteriorate, such as cracking. Asphalt pavement materials can also deteriorate, such as becoming chipped. The deterioration of such road surfaces has unique characteristics for each location. Therefore, it is possible to identify the location where a camera image was taken, based on these unique characteristics.

[0043] In this embodiment, camera images (hereinafter referred to as road surface camera images) taken in advance of the road surface at a plurality of registered points are associated with the position information of each registered point and registered as road surface registration information in an image position DB (database) constructed in the roadside device 3 (see FIG. 4). Meanwhile, in the pedestrian terminal 1, the camera 11 takes an image of the road surface at the feet of the pedestrian and outputs the foot camera image in real time. Then, as shown in FIG. 2, the pedestrian terminal 1 executes an image matching process for matching the road surface camera images of the registered points registered in the image position DB with the real-time foot camera images output from the camera 11, and acquires the position information corresponding to the road surface camera images of the registered points where matching has been successful as the position information of the pedestrian's current location.

[0044] In this embodiment, as shown in Fig. 3, when the roadside unit 3 is installed at an intersection, the surrounding area of ​​the intersection where the roadside unit 3 is installed, specifically, the intersection where the roadside unit 3 is installed and a predetermined section of the road connected to the intersection, become the target range of the image position DB. Within this target range, registered points are set at predetermined intervals (e.g., 25 cm). Note that pedestrians usually walk on the crosswalk at the intersection, or on the sidewalk or shoulder of the road connected to the intersection, so registered points are set on the road where the pedestrian walks (i.e., the walking route).

[0045] 2, when a pedestrian walks across a crosswalk at an intersection or on a sidewalk or shoulder of a road, a real-time foot camera image of the road surface under the pedestrian's feet is periodically output from the camera in the pedestrian terminal 1. The pedestrian terminal 1 performs an image matching process using the real-time foot camera image, and when the pedestrian reaches a registered point, if the image matching is successful, the current location of the pedestrian is identified.

[0046] Also, the moving direction of the pedestrian is obtained by the image matching process. That is, if the orientation of the camera image, for example, the upward direction of the camera image (east-west-north-south), is registered in the image position DB, the camera image registered in the image position DB is rotated by the image matching process to match the orientation with the real-time foot camera image output from the camera 11, and the upward direction of the real-time foot camera image, that is, the moving direction of the pedestrian is obtained.

[0047] In the image matching process, if many (or all) camera images registered in the image position DB are used as matching candidates for the real-time foot camera image, rapid processing (real-time processing) may be difficult. Therefore, when executing the image matching process, the pedestrian terminal 1 performs a matching candidate extraction process to extract camera images (hereinafter referred to as matching candidate images) that are more suitable as targets for matching the real-time foot camera image from the camera images registered in the image position DB.

[0048] In the matching candidate extraction process, the pedestrian terminal 1 performs tentative positioning of the pedestrian W. As a result, the pedestrian terminal 1 acquires rough position information (hereinafter referred to as tentative positioning information) of the current location of the pedestrian W. In general, the tentative positioning information has a lower accuracy than the position information obtained by the image matching process.

[0049] In this embodiment, three-dimensional map information (fixed object registration information) is registered in a three-dimensional map DB (database) previously constructed on the cloud. This three-dimensional map information is obtained by associating visual feature information (here, a plurality of feature points) of the contours of fixed objects (structures such as buildings and bridges) included in a camera image (surrounding camera image) that captures the surroundings of the road surface on the pedestrian's walking route with their position information. The pedestrian terminal 1 extracts feature information (here, a plurality of feature points) regarding objects (including fixed objects such as buildings) included in a front camera image output in real time from the camera 11, and performs provisional positioning of the pedestrian by collating (comparing) the feature information regarding the objects with the feature information regarding the fixed objects included in the three-dimensional map information. For such provisional positioning by the pedestrian terminal 1, known technologies such as Area Learning and VPS (Visual Positioning Service) can be adopted.

[0050] 5, when a pedestrian W moves in the direction indicated by arrow D, the pedestrian terminal 1, upon reaching a position where a building A (object) as a fixed object can be photographed by the camera 11, performs provisional positioning of the pedestrian W by comparing multiple feature points (here, parts indicated by black circles) of the building A included in a real-time forward camera image with multiple feature points included in the three-dimensional map information. The pedestrian terminal 1 specifies a provisional current location of its own device (i.e., the pedestrian W) based on the result of the provisional positioning (i.e., provisional positioning information), and extracts a camera image capturing the road surface near the provisional current location as a candidate image to be matched.

[0051] Furthermore, when extracting a matching candidate image, the pedestrian terminal 1 can acquire in advance a part of the three-dimensional map information (fixture registration information) registered in the three-dimensional map DB on the cloud and store the acquired information in the memory of the pedestrian terminal 1. For example, the pedestrian terminal 1 may receive a satellite positioning signal and acquire only the three-dimensional map information required for tentative positioning (i.e., corresponding to the area around the pedestrian W) from the three-dimensional map DB based on the rough position information of the current location of the pedestrian W specified based on the satellite positioning signal. This allows the pedestrian terminal 1 to reduce the time required to acquire the three-dimensional map information and the amount of data of the three-dimensional map information stored in the memory of the pedestrian terminal 1. The three-dimensional map DB from which the pedestrian terminal 1 acquires the three-dimensional map information may be provided in any other device (e.g., a computer such as a server) that can communicate with the pedestrian terminal 1 via a communication network.

[0052] Furthermore, as the pedestrian W moves, buildings B and C as fixed objects are included in the real-time forward camera image output from the camera 11. As in the case of building A described above, the pedestrian terminal 1 can sequentially perform tentative positioning of the pedestrian W by using a plurality of characteristic points (here, the areas indicated by black circles) of buildings B and C included in the real-time forward camera image.

[0053] The 3D map information is not limited to a 3D map DB constructed on a cloud, but may be stored in a memory provided in a roadside device 3 located in the vicinity of the walking route. In this case, the roadside device 3 only needs to store 3D map information related to the area surrounding the device itself.

[0054] The pedestrian terminal 1 can also perform look-ahead image matching. In this look-ahead image matching, the next registered point that the pedestrian W will reach is predicted based on the pedestrian's past positions (for example, the registered point where the previous image matching was performed or the pedestrian's position obtained by the previous tentative positioning) and the progress of the pedestrian W obtained from the detection results of the acceleration sensor 12 and the gyro sensor 13 (see FIG. 6). Based on the prediction result, a foot camera image to be matched is extracted from the foot camera images registered in the image position DB, and the extracted foot camera image is matched with the real-time foot camera image output from the camera 11.

[0055] For example, as the progress state of the pedestrian W, the progress direction of the pedestrian W measured based on the detection result of the gyro sensor 13 and the movement speed of the pedestrian W measured based on the detection result of the acceleration sensor 12 are obtained, and the registration point that the pedestrian W will reach next is predicted based on the progress direction and movement speed of the pedestrian. Note that the registration point that the pedestrian W will reach next can also be predicted based only on the progress direction of the pedestrian W. In this case, the registration point located ahead in the progress direction of the pedestrian W is selected as the registration point that the pedestrian W will reach next.

[0056] In the look-ahead image matching, a process is performed to predict the next registered point that the pedestrian will reach based on the pedestrian's progress. At this time, PDR (Pedestrian Dead Reckoning) positioning is performed to estimate the pedestrian's current location, and the next registered point that the pedestrian will reach may be predicted based on the result of the PDR positioning.

[0057] Such pre-read image matching can speed up the image matching process in the pedestrian terminal 1. Also, the processing load on the processor involved in image matching is reduced.

[0058] Next, a description will be given of the schematic configuration of the pedestrian terminal 1 and the roadside unit 3 according to the first embodiment.

[0059] The pedestrian terminal 1 includes a camera 11, an acceleration sensor 12, a gyro sensor 13, a satellite positioning unit 14, an ITS communication unit 15, a wireless communication unit 16, a memory 17, and a processor 18.

[0060] The camera 11 includes a foot camera 11A that captures an image of the feet of a pedestrian, and a front camera 11B that captures an image in front of the pedestrian.

[0061] The acceleration sensor 12 detects the acceleration occurring in the body of the pedestrian. The gyro sensor 13 detects the angular velocity occurring in the body of the pedestrian. Note that the pedestrian terminal 1 may be provided with other motion sensors.

[0062] The satellite positioning unit 14 includes a receiver that receives satellite positioning signals from satellites in a satellite positioning system such as GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System), etc. The satellite positioning unit 14 measures the position of the device itself from the received satellite positioning signals, and acquires position information (latitude and longitude) of the device itself as position information of a pedestrian.

[0063] The ITS communication unit 15 broadcasts messages to the in-vehicle terminal 2 and the roadside unit 3 through ITS communication (pedestrian-to-vehicle communication and road-to-pedestrian communication), and also receives messages transmitted from the in-vehicle terminal 2 and the roadside unit 3.

[0064] The wireless communication unit 16 transmits messages to the roadside unit 3 by wireless communication such as WiFi (registered trademark), and receives messages transmitted from the roadside unit 3. The ITS communication unit 15 and the wireless communication unit 16 each have known hardware such as an antenna and a communication circuit for communicating with other devices.

[0065] The memory 17 stores map information, programs executed by the processor 18, and the like. In this embodiment, the memory 17 stores road registration information acquired from the image position DB, that is, camera images and position information for each registration point. The memory 17 also stores three-dimensional map information acquired from the three-dimensional map DB, that is, feature information and position information related to fixed objects such as buildings located around the road surface on the walking route. In this embodiment, when the pedestrian terminal 1 approaches an intersection, it acquires road registration information of the image position DB related to the surrounding area of ​​the intersection from the roadside device 3 installed at the intersection. In addition, by constructing a three-dimensional map DB in the roadside device 3, the pedestrian terminal 1 may acquire three-dimensional map information of the three-dimensional map DB related to the surrounding area from the roadside device 3.

[0066] The processor 18 performs various processes by executing programs stored in the memory 17. In this embodiment, the processor 18 performs message control process, collision determination process, attention-call control process, speed measurement process, direction measurement process, tentative positioning process, match candidate extraction process, image matching process, and position information acquisition process. Note that the pedestrian terminal 1 may execute various processes by multiple processors. Also, the pedestrian terminal 1 may acquire the processing results by having another information processing device execute part of the above processes.

[0067] In the message control process, the processor 18 controls the transmission and reception of messages of ITS communication between the vehicle-mounted terminal 2 and the roadside unit 3. The processor 18 also controls the transmission and reception of messages of wireless communication with the roadside unit 3.

[0068] In the collision determination process, the processor 18 determines whether there is a risk of the vehicle colliding with a pedestrian based on the vehicle position information contained in the vehicle information obtained from the in-vehicle terminal 2 and the pedestrian position information obtained by the satellite positioning unit 14.

[0069] In the attention-attraction control process, the processor 18 performs control so as to perform a predetermined attention-attraction action (such as audio output or vibration) for pedestrians when it is determined in the collision determination process that there is a risk of collision.

[0070] In the speed measurement process, the processor 18 measures the speed of the pedestrian based on the detection result of the acceleration sensor 12. When the pedestrian walks, acceleration occurs in the body of the pedestrian, and the walking pitch of the pedestrian is obtained based on the change in this acceleration. In addition, the speed is calculated from the walking pitch and stride length. Note that the stride length may be set based on the attributes of the pedestrian registered in the pedestrian terminal 1 (adult, child, etc.).

[0071] In the direction measurement process, the processor 18 measures the moving direction of the pedestrian based on the detection result of the gyro sensor 13.

[0072] In the tentative positioning process, the processor 18 compares feature information on an object included in a real-time front camera image output from the front camera 11B with feature information on fixed objects such as buildings included in the three-dimensional map information. The processor 18 obtains tentative positioning information based on the comparison result.

[0073] In the matching candidate extraction process, processor 18 extracts matching candidate images suitable for matching with real-time foot camera images from camera images registered in the image position DB based on the provisional positioning information. More specifically, processor 18 extracts road surface camera images included in road surface registration information related to registered points around the approximate current location of the pedestrian in the provisional positioning information as matching candidate images.

[0074] In addition, in the match candidate extraction process, processor 18 may predict the next registered point (predicted point) that the pedestrian will reach based on the pedestrian's past position, movement speed, and traveling direction, and extract a match candidate image based on the prediction result.

[0075] In the image matching process, the processor 18 matches the real-time foot camera image output from the camera 11 with the match candidate image extracted in the match candidate extraction process. At this time, the processor 18 extracts feature information (information on feature points) from each of the real-time foot camera image and the match candidate image, and compares each piece of feature information to match the camera images. This image matching process may be performed using AI (artificial intelligence).

[0076] In the position information acquisition process, the processor 18 acquires, as position information of the pedestrian's current location, position information associated with the camera image of the registered point that has been successfully matched in the image matching process.

[0077] The vehicle-mounted terminal 2 is also equipped with a processor and memory (not shown) and can execute programs stored in the memory to perform message control processing, collision determination processing, and warning control processing similar to those of the pedestrian terminal 1.

[0078] The roadside unit 3 includes an ITS communication unit 31, a wireless communication unit 32, a memory 33, and a processor .

[0079] The ITS communication unit 31 broadcasts messages to the pedestrian terminal 1 and the vehicle-mounted terminal 2 through ITS communication (road-to-pedestrian communication, road-to-vehicle communication), and also receives messages transmitted from the pedestrian terminal 1 and the vehicle-mounted terminal 2.

[0080] The wireless communication unit 32 transmits messages to the pedestrian terminal 1 by wireless communication such as WiFi (registered trademark), and receives messages transmitted from the pedestrian terminal 1. The ITS communication unit 31 and the wireless communication unit 32 each have known hardware such as an antenna and a communication circuit for communicating with other devices.

[0081] The memory 33 stores programs executed by the processor 34. In this embodiment, the memory 33 stores road surface registration information of an image position DB (see FIG. 4). The memory 33 may also store three-dimensional map information of a three-dimensional map DB.

[0082] The processor 34 performs various processes by executing programs stored in the memory 33. In this embodiment, the processor 34 performs a message control process and an image position DB management process.

[0083] In the message control process, the processor 34 controls the transmission and reception of messages of ITS communication between the pedestrian terminal 1 and the in-vehicle terminal 2. The processor 34 also controls the transmission and reception of messages of wireless communication with the pedestrian terminal 1.

[0084] In the image position DB management process, the processor 34 manages the image position DB (see FIG. 4). In this image position DB, camera images and position information for each registered point are registered. In this embodiment, the registration information of the image position DB is delivered to the pedestrian terminal 1 in response to a request from the pedestrian terminal 1.

[0085] Next, the operation procedures of the pedestrian terminal 1, the vehicle-mounted terminal 2, and the roadside unit 3 according to the first embodiment will be described. Fig. 7 and Fig. 8 are flow diagrams showing the operation procedures of the pedestrian terminal 1. Fig. 9 is a flow diagram showing the operation procedures of the vehicle-mounted terminal 2. Fig. 10 is a flow diagram showing the operation procedures of the roadside unit 3.

[0086] 7(A), in the pedestrian terminal 1, first, the satellite positioning unit 14 acquires the position information of the pedestrian (ST101). Next, the processor 18 determines whether or not the situation is such that the pedestrian information should be transmitted, specifically, whether or not the user has entered a dangerous area, based on the position information of the pedestrian (ST102).

[0087] Here, if the situation is such that pedestrian information should be transmitted (Yes in ST102), in response to a transmission instruction from the processor 18, the ITS communication unit 15 transmits an ITS communication message including pedestrian information (such as pedestrian ID and location information) to the in-vehicle terminal 2 and the roadside unit 3 (ST103).

[0088] As shown in FIG. 9, when the in-vehicle terminal 2 receives an ITS communication (pedestrian-to-vehicle communication) message from the pedestrian terminal 1 (Yes in ST201), it performs a collision judgment to determine whether there is a risk of the vehicle colliding with a pedestrian based on the vehicle position information and other information contained in the message (ST202).

[0089] Here, if there is a risk of the vehicle colliding with a pedestrian (Yes in ST202), the in-vehicle terminal 2 performs a predetermined attention-calling operation for the driver (ST203). Specifically, the in-vehicle terminal 2 causes the car navigation device to perform an attention-calling operation (for example, voice output or screen display) as the attention-calling operation. Note that, if the vehicle is an autonomous vehicle, the in-vehicle terminal 2 instructs the autonomous driving ECU (cruising control device) to perform a predetermined collision avoidance operation.

[0090] 10(A), in the roadside unit 3, when the ITS communication unit 31 receives a message of ITS communication (pedestrian-to-vehicle communication) from the pedestrian terminal 1 (Yes in ST301), the processor 34 acquires the terminal ID and position information of the pedestrian terminal 1 included in the received message (ST302). Next, the processor 34 determines whether or not the pedestrian terminal 1 is located in the vicinity of the target area of ​​the registration information of the image position DB (inside or near the target area) based on the pedestrian's position information (ST303).

[0091] Here, if the pedestrian terminal 1 is located in the vicinity of the target area (Yes in ST303), in response to a transmission instruction from processor 34, ITS communication unit 31 transmits an ITS communication message to pedestrian terminal 1 including DB usage information indicating that the pedestrian terminal 1 can use the registration information in its own database (memory 17) (ST304).

[0092] 7(B), in the pedestrian terminal 1, when the ITS communication unit 15 receives an ITS communication message including DB usage information from the roadside unit 3 (Yes in ST111), the wireless communication unit 16 transmits a wireless communication message requesting DB registration information (here, road surface registration information of the image position DB) to the roadside unit 3 in response to a transmission instruction from the processor 18 (ST112). Note that the DB registration information may include 3D map information of the 3D map DB.

[0093] As shown in FIG. 10(B), in the roadside unit 3, when the wireless communication unit 32 receives a wireless communication message requesting DB registration information from the pedestrian terminal 1 (Yes in ST311), in response to a transmission instruction from the processor 34, the wireless communication unit 32 transmits a wireless communication message including the DB registration information to the pedestrian terminal 1 (ST312).

[0094] At this time, all road surface registration information in the image position DB of the roadside device 3 may be transmitted to the pedestrian terminal 1, or only a portion of the road surface registration information that is likely to be used by the pedestrian terminal 1 may be transmitted to the pedestrian terminal 1. Specifically, road surface registration information within a predetermined range around the pedestrian terminal 1, particularly within a predetermined range located in the traveling direction of the pedestrian, may be transmitted to the pedestrian terminal 1.

[0095] As shown in FIG. 7(C), in the pedestrian terminal 1, when the wireless communication unit 16 receives a wireless communication message including DB registration information from the roadside unit 3 (Yes in ST121), the processor 18 registers the DB registration information included in the received message in the database of the device itself (ST122).

[0096] Next, as shown in FIG. 8, in the pedestrian terminal 1, the processor 18 acquires position information of the pedestrian (ST131). This position information of the pedestrian (provisional positioning information) is obtained as a result of the provisional positioning by the pedestrian terminal 1. More specifically, the processor 18 extracts feature information on objects included in the forward camera image output in real time from the camera 11, and compares the feature information with feature information on fixed objects such as buildings included in the three-dimensional map information, thereby acquiring the position information of the pedestrian (absolute position information). In step ST131, the processor 18 may acquire the position information of the pedestrian based on the received satellite positioning signal instead of executing the provisional positioning. The position information of the pedestrian acquired in step ST131 is sequentially stored in the memory 17.

[0097] Note that the pedestrian terminal 1 can obtain 3D map information of the range required for the provisional positioning in step ST131 from a 3D map DB constructed on the cloud, and store it in its own database.

[0098] The processor 18 also acquires a real-time foot camera image of the pedestrian's feet captured by the camera 11 (ST132). This real-time foot camera image is repeatedly acquired at a predetermined time interval. The processor 18 also measures the moving speed of the pedestrian based on the detection result of the acceleration sensor 12 (ST133). The processor 18 also measures the moving direction (moving direction) of the pedestrian based on the detection result of the gyro sensor 13 (ST134).

[0099] Next, as a match candidate extraction process, processor 18 extracts match candidate images to be matched with the real-time foot camera image acquired in step ST132 based on the pedestrian's position information (i.e., the tentative current location) acquired in step ST131 (ST135).

[0100] More specifically, in step ST135, the processor 18 can extract road camera images corresponding to registered points located within a predetermined range from the pedestrian's tentative current location as match candidate images. At this time, the processor 18 may limit the match candidate images to be extracted to those corresponding to registered points located approximately in front of and behind the pedestrian (in the forward and reverse directions).

[0101] Next, as an image matching process, processor 18 matches the match candidate image extracted from the database of its own device with the real-time foot camera image output from camera 11 (ST136).

[0102] In step ST135, processor 18 may predict a registration point (predicted point) where the pedestrian will next arrive based on the pedestrian's past position (for example, the pedestrian's position obtained by the previous tentative positioning), the moving speed, and the traveling direction, and extract a match candidate image based on the prediction result. In this way, processor 18 can match the extracted match candidate image with the real-time foot camera image output from camera 11 (preview image matching) at the timing when the pedestrian arrives at the predicted point in step ST136.

[0103] If the image matching process is successful, that is, if the match candidate image extracted from the database of the own device matches the real-time foot camera image (Yes in ST137), the processor 18 performs position information acquisition process to acquire position information associated with the road camera image of the registered point where matching was successful as position information of the pedestrian's current location (ST138). The pedestrian terminal 1 can repeatedly execute the above steps ST131 to ST138.

[0104] In this embodiment, the roadside device 3 provides the road surface camera image of the registered point to the pedestrian terminal 1, and the image matching process is performed in the pedestrian terminal 1. However, the roadside device 3 may provide the pedestrian terminal 1 with feature information (information on feature points) extracted from the road surface camera image of the registered point. In this case, in the image matching process, the pedestrian terminal 1 compares the feature information of the registered point acquired from the roadside device 3 with the feature information extracted from the real-time foot camera image. Also, an image of a characteristic part may be cut out from the road surface camera image of the registered point, and the image matching process may be performed using the image of the characteristic part. In this way, the capacity of the road surface registration information in the image position DB delivered from the roadside device 3 to the pedestrian terminal 1 can be reduced, and the load of wireless communication between the roadside device 3 and the pedestrian terminal 1 can be reduced.

[0105] Further, all communication means between the roadside unit 3 and the pedestrian terminal 1 may be cellular communication, and the functions of the roadside unit 3 may be placed on the cloud, so that the image position DB can be managed over a wider range.

[0106] In this way, the pedestrian terminal 1 compares the match candidate image extracted from the road camera image based on the provisional positioning result with the foot camera image, and obtains the position information of the registered point corresponding to the successfully matched match candidate image as the position information of the pedestrian's current location, thereby reducing the processing load of the processor 18 when pedestrian positioning is performed using camera images (foot camera image and road camera image) capturing the road surface on which the pedestrian is walking.

[0107] Second embodiment Next, a traffic safety support system according to a second embodiment will be described. FIG. 11 is a block diagram showing a schematic configuration of a pedestrian terminal 1 according to the second embodiment. FIG. 12 is a flow diagram showing an operation procedure of the pedestrian terminal 1 according to the second embodiment. Note that, with respect to the second embodiment, matters that are not particularly mentioned below are the same as those in the first embodiment. Also, in the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0108] In the second embodiment, as shown in Fig. 11, the processor 18 performs a relative positioning process in addition to various processes in the first embodiment. In the relative positioning process, the processor 18 sequentially calculates the amount of movement of the pedestrian from a reference position (here, the latest current position of the pedestrian acquired by the position information acquisition process) based on pedestrian autonomous navigation (PDR) using the acceleration sensor 12 and the gyro sensor 13, and acquires new position information of the current position of the pedestrian by accumulating the amount of movement. In the relative positioning process, the processor 18 can also sequentially calculate the amount of movement of the pedestrian from a reference position by executing self-position estimation and environmental map creation using the camera 11 (front camera 11B) based on a known Visual SLAM (Simultaneous Localization and Mapping) technology.

[0109] In the second embodiment, as shown in Fig. 12, the processor 18 executes steps ST431 to ST438 (hereinafter referred to as positioning based on an absolute position) similar to steps ST131 to ST138 shown in Fig. 8. In the positioning based on an absolute position, position information of a pedestrian's current location is acquired by repeatedly performing tentative positioning (positioning of an absolute position), so that the processing load of the processor 18 is relatively high.

[0110] Therefore, in the second embodiment, the processor 18 further suspends positioning based on the absolute position as long as the positioning error is within a predetermined range, and performs positioning based on the relative position, which has a relatively small processing load, in the meantime to obtain position information of the pedestrian's current location.

[0111] In the positioning based on the relative position, first, the processor 18 calculates the amount of movement and the direction of the pedestrian from the reference position of the pedestrian by the relative positioning process, and acquires new position information of the current location of the pedestrian based on the calculation result of the amount of movement and the direction (ST439). Here, as the reference position of the pedestrian, the current location of the pedestrian acquired in step ST438 is used immediately after the transition from the positioning based on the absolute position to the positioning based on the relative position, and thereafter, the new location calculated by the previous positioning based on the relative position is used.

[0112] Next, similar to the matching candidate extraction process in step ST436, processor 18 extracts a matching candidate image to be matched with the real-time foot camera image based on the new position acquired in step ST439 (ST440). In this way, while the tentative positioning based on the absolute position is suspended, processor 18 can sequentially calculate the amount of movement and direction of the pedestrian, perform tentative positioning based on the calculation results of the amount of movement and direction, and extract some of the multiple road camera images stored in memory as matching candidate images.

[0113] Next, as an image matching process, processor 18 matches a match candidate image extracted from the database of its own device with the latest foot camera image output from camera 11 (ST441).

[0114] If the image matching process is successful (Yes in ST442), processor 18 performs a location information acquisition process to acquire location information associated with the road camera image of the registered point where matching was successful as location information of the pedestrian's current location (ST443).

[0115] Thereafter, the processor 18 determines whether or not the measurement error due to the relative position is within the allowable range, and if it is within the allowable range (Yes in ST444), the processor 18 continues the positioning based on the relative position by returning to step ST439. On the other hand, if the measurement error due to the relative position exceeds the allowable range (No in ST444), the processor 18 ends the positioning based on the relative position by returning to step ST431, and performs positioning based on the absolute position again.

[0116] Third embodiment Next, a traffic safety support system according to the third embodiment will be described. FIG. 13 is a block diagram showing a schematic configuration of the pedestrian terminal 1 according to the third embodiment. FIG. 14 is a flow diagram showing an operation procedure of the pedestrian terminal 1 according to the third embodiment. Note that, with respect to the third embodiment, matters that are not particularly mentioned below are similar to those in the first or second embodiment. Also, in the third embodiment, the same components as those in the first or second embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0117] In the third embodiment, as shown in Fig. 13, the processor 18 performs an occlusion detection process in addition to the various processes in the first embodiment. In the occlusion detection process, the processor 18 determines whether or not an occlusion has occurred in the shooting area of ​​the front camera 11B. This allows the pedestrian terminal 1 to avoid using an inappropriate front camera image when an occlusion has occurred in the shooting area of ​​the front camera 11B, and to stably acquire position information of the current location of the pedestrian.

[0118] In the third embodiment, as shown in Fig. 14, the processor 18 acquires the position information of the pedestrian in the same manner as in step ST131 shown in Fig. 8 (ST531). Next, the processor 18 detects whether the front camera 11B is occluded by an occlusion detection process, and if occlusion occurs (Yes in ST532), acquires the position information of the pedestrian that was normally acquired in the previous process of step ST531 as the current position information of the pedestrian (i.e., the latest tentative positioning result) (ST533). This makes it possible for the pedestrian terminal 1 to avoid using an inappropriate front camera image when occlusion occurs in the shooting area of ​​the front camera 11B, and to stably acquire the position information of the current location of the pedestrian.

[0119] In step ST532, the processor 18 can determine that the front camera 11B is obscured, for example, when acquisition of the position information of the pedestrian in step ST531 has failed or when there is an abnormality in the position information. Alternatively, the processor 18 can determine that the front camera 11B is obscured, when an obstruction (for example, an object that may hinder execution of Area Learning, VPS, or the like) is detected in the front camera image based on a known technique.

[0120] After that, the processor 18 executes steps ST534 to ST540 which are similar to steps ST132 to ST138 shown in FIG.

[0121] When one omnidirectional camera (360-degree camera) is used as the camera 11, a partial image area in the camera image obtained by shooting with the omnidirectional camera is used as a front camera image in step ST531. Also, a partial image area in the camera image obtained by shooting with the omnidirectional camera is used as a foot camera image in step ST534.

[0122] Furthermore, if obstruction occurs in step ST532 (Yes in ST532), the pedestrian terminal 1 returns to step ST531, acquires a camera image (or a partial image area) captured by the omnidirectional camera in a direction other than the front of the pedestrian that is not affected by the obstruction, and can use the camera image in the direction other than the front as a front camera image. In that case, step ST533 can be omitted. The above-mentioned omnidirectional camera can be similarly applied to other embodiments and modified examples of the present disclosure.

[0123] (First modified example of the third embodiment) Next, a traffic safety support system according to a first modified example of the third embodiment will be described. Fig. 15 is a flow diagram showing the operation procedure of the pedestrian terminal 1 according to the first modified example of the third embodiment. Note that, regarding the first modified example, matters that are not particularly mentioned below are the same as those in the third embodiment. Also, in the first modified example, the same components as those in the third embodiment are denoted by the same reference numerals, and detailed descriptions thereof will be omitted.

[0124] In the first modification, as shown in Fig. 15, the processor 18 acquires position information of a pedestrian and detects whether or not the front camera is blocked (ST631, ST632) in the same manner as in steps ST531 and ST532 shown in Fig. 14. When the processor 18 detects that the front camera is blocked (Yes in ST632), it determines whether or not the duration of the blocking is equal to or longer than a preset threshold (ST633).

[0125] Therefore, when the duration of the obstruction is equal to or less than a preset threshold (No in ST633), the processor 18 acquires the position information of the pedestrian normally acquired in the previous process of step ST631 as the current position information of the pedestrian (ST634), similar to step ST533 in Fig. 14. On the other hand, when the duration of the obstruction is equal to or more than the threshold (Yes in ST633), the processor 18 acquires new position information of the current location of the pedestrian by calculating the amount of movement of the pedestrian from the reference position of the pedestrian through the relative positioning process, similar to step ST439 in Fig. 12 (ST635). Here, when the relative positioning in step ST635 is executed for the first time, the position of the pedestrian acquired in the previous step ST634 is used as the reference position of the pedestrian, and thereafter, the new position calculated by the previous relative positioning is used.

[0126] Subsequently, the processor 18 executes steps ST636 to ST642 similar to steps ST534 to ST540 shown in Fig. 14. As a result, when the duration of the occlusion occurring in the photographing area of ​​the front camera 11B becomes long, the processor 18 can appropriately acquire the position information of the current location based on the amount of movement of the pedestrian. In addition, in the matching candidate extraction process of step ST639, the processor 18 can use either the position acquired in step ST634 or ST635 depending on the result of the determination (ST633) as to whether the duration of the occlusion is equal to or longer than a preset threshold value.

[0127] (Second modified example of the third embodiment) Next, a traffic safety support system according to a second modified example of the third embodiment will be described. Fig. 16 is a flow diagram showing the operation procedure of the pedestrian terminal 1 according to the second modified example of the third embodiment. Note that, with respect to the second modified example, matters that are not particularly mentioned below are the same as those in the third embodiment. In addition, in the second modified example, the same components as those in the third embodiment are denoted by the same reference numerals, and detailed descriptions thereof will be omitted.

[0128] In the second modification, as shown in Fig. 16, the processor 18 acquires the position information of the pedestrian and detects whether the front camera is blocked (ST731, ST732) in the same manner as steps ST531 and ST532 shown in Fig. 14. When the processor 18 (ITS communication unit 15) detects that the front camera is blocked (Yes in ST732), it transmits an ITS communication message including the pedestrian information (pedestrian ID and position information, etc.) to the vehicle and roadside unit by pedestrian-to-vehicle communication and pedestrian-to-path communication, respectively (ST733). This allows the in-vehicle device to be notified of the situation in which the front of the pedestrian is blocked, thereby improving the safety of the pedestrian and the vehicle.

[0129] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0130] The pedestrian device and its positioning method according to the present disclosure have the effect of reducing the processing load when pedestrian positioning is performed using camera images capturing the road surface on which the pedestrian is walking, and are useful as a pedestrian device and its positioning method that is carried by a pedestrian and performs positioning to obtain the pedestrian's position information. [Explanation of symbols]

[0131] 1 Pedestrian terminal (pedestrian device) 2. Vehicle-mounted terminal (vehicle-mounted device) 3 Roadside unit 11 Camera 11A Foot Camera 11B Front camera 12 Acceleration Sensor 13 Gyro sensor 14 Satellite positioning unit 15 ITS Communication Department 16 Wireless Communication Section 17 Memory 18 Processors 31 ITS Communication Department 32 Wireless Communication Section 33 Memory 34 processors

Claims

1. a foot camera that sequentially generates foot camera images by photographing the road surface under the feet of a pedestrian; A surrounding camera that captures at least one of front, rear, left, and right of the pedestrian to sequentially generate pedestrian surrounding camera images; a memory for storing road surface registration information in which position information of a registered point is associated with a plurality of road surface camera images of the road surface at the registered point, and fixed object registration information in which position information is associated with feature information on fixed objects included in surrounding camera images of the surrounding area; a processor for executing a process of acquiring location information of a current location of the pedestrian; The processor, Extracting feature information regarding an object included in the pedestrian surroundings camera image; performing provisional positioning of the pedestrian using the position information in the fixed object registration information by comparing the characteristic information of the object with the characteristic information of the fixed object in the fixed object registration information; extracting a part of the plurality of road camera images stored in the memory as a match candidate image based on a result of the provisional positioning; Matching the foot camera image with each of the match candidate images; The pedestrian device acquires position information of the registered point associated with the match candidate image for which matching has been successful as position information of the pedestrian's current location.

2. A receiver for receiving a satellite positioning signal is further provided, The pedestrian device according to claim 1 , wherein the processor acquires the fixed object registration information from another device in response to position information obtained from the satellite positioning signal, and stores the acquired fixed object registration information in the memory.

3. The pedestrian device according to claim 1 , wherein the foot camera and the surrounding camera are composed of a single 360-degree camera.

4. The processor, The provisional positioning may be interrupted, Sequentially calculating the amount of movement of the pedestrian while the provisional positioning is suspended; The pedestrian device according to claim 1 , wherein a part of the plurality of road camera images stored in the memory is extracted as the match candidate image based on a result of the calculation of the amount of movement.

5. 5. The pedestrian device according to claim 4, wherein the processor acquires the amount of movement of the pedestrian by at least one of a self-position estimation process based on pedestrian autonomous navigation and a self-position estimation process based on a pedestrian surrounding camera image generated by the surrounding camera.

6. The processor, repeatedly performing the provisional position determination of the pedestrian; storing the results of the provisional positioning in the memory in sequence; determining whether or not an occlusion has occurred in a shooting area of ​​the surrounding camera based on the pedestrian surrounding camera image; The pedestrian device according to claim 1 , wherein, when it is determined that an obstruction has occurred in the imaging area of ​​the surrounding camera, the past provisional positioning results stored in the memory are acquired as the latest provisional positioning result.

7. The processor, The provisional positioning may be interrupted, Sequentially calculating the amount of movement of the pedestrian while the provisional positioning is suspended; extracting a part of the plurality of road camera images stored in the memory as the matching candidate image based on a result of the calculation of the amount of movement; When it is determined that the occlusion has occurred in the photographing area of ​​the surrounding camera, a duration of the occlusion is calculated; 7. The pedestrian device according to claim 6, wherein, when a duration of the occlusion becomes equal to or longer than a threshold, the tentative positioning is interrupted, and an amount of movement of the pedestrian is acquired by at least one of a self-position estimation process based on pedestrian autonomous navigation and a self-position estimation process based on the pedestrian surrounding camera images generated by the surrounding camera.

8. A communication unit that performs wireless communication with at least one of an in-vehicle device mounted in the vehicle and a roadside device, 8. The pedestrian device according to claim 6, wherein when the processor determines that an occlusion has occurred in the shooting area of ​​the surrounding camera, the processor transmits a message regarding the occurrence of the occlusion to at least one of the in-vehicle device and the roadside device that perform the wireless communication via the communication unit.

9. A positioning method for a pedestrian device for acquiring current position information of a pedestrian, comprising: By taking pictures of the road surface under the feet of pedestrians with a camera, the system sequentially generates foot camera images. Further, by photographing at least one of the front, rear, left and right of the pedestrian with a camera, pedestrian surrounding camera images are sequentially generated; road surface registration information in which position information of a registered point is associated with a plurality of road surface camera images of the road surface at the registered point, and fixed object registration information in which position information of a fixed object is associated with feature information of the fixed object included in surrounding camera images of the surrounding area, are stored in a memory; Extracting feature information regarding an object included in the pedestrian surroundings camera image; performing provisional positioning of the pedestrian using the position information in the fixed object registration information by comparing the characteristic information of the object with the characteristic information of the fixed object in the fixed object registration information; extracting a part of the plurality of road camera images stored in the memory as a match candidate image based on a result of the provisional positioning; Matching the foot camera image with each of the match candidate images; The positioning method for a pedestrian device further comprises acquiring position information of the registered point associated with the match candidate image for which matching has been successful as position information of the pedestrian's current location.

Images