Pedestrian device, positioning system, and positioning method

Abstract

To obtain highly accurate positional information in implementation of visual positioning and achieve power saving by reducing power consumption.SOLUTION: A pedestrian terminal 1 includes: a camera 12 for photographing surroundings of a pedestrian; a state sensor 13 for detecting a state of the pedestrian; a memory 17 for storing registration information of a peripheral environment DB related to a peripheral environment of the pedestrian; and a processor 18. The processor always comprises: obtaining positional information of the pedestrian in implementation of satellite positioning; implementing visual positioning for estimating a current position of the pedestrian by collating an image captured by the camera with a candidate image registered in the peripheral environment information DB when a predetermined change appears in the state of the pedestrian, on the basis of detection results of the state sensor; and controlling ON / OFF of the camera in accordance with implementation timing of the visual positioning.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a pedestrian device that is held by a pedestrian to measure the current position of the pedestrian, a mobile device that is held by a mobile body such as a pedestrian or a vehicle to measure the current position of the mobile body, a positioning system, and a positioning method. [Background technology]

[0002] In a safe driving support wireless system that uses ITS (Intelligent Transport System), vehicle location information is exchanged between in-vehicle terminals to avoid accidents between vehicles, and vehicle-pedestrian accidents are avoided by exchanging vehicle and pedestrian location information between in-vehicle terminals and pedestrian terminals.

[0003] In-vehicle and pedestrian terminals mainly obtain the position information of vehicles and pedestrians by satellite positioning, but various positioning methods can also be adopted, such as positioning using Pedestrian Dead Reckoning (PDR). In this case, it is desirable to adopt high-precision positioning technology in order to prevent traffic accidents.

[0004] One such highly accurate positioning technology is known as visual positioning, for example, a technology called VPS (Visual Positioning Service) (see Patent Document 1). In this technology, a positioning device compares a captured image output from a camera in real time with a candidate image registered in a database, and if the two match, acquires the location information associated with the candidate image as the location information of the current location of the moving object. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2012 / 086821 Summary of the Invention [Problem to be solved by the invention]

[0006] With satellite positioning, the accuracy of positioning decreases due to the effects of multipath and the blockage of satellite radio waves caused by pedestrians entering the shadows of buildings. On the other hand, visual positioning can ensure high accuracy, but it consumes a lot of power because it requires the operation of a camera and a large load of matching processing. For this reason, if visual positioning is performed frequently, there is a problem that the pedestrian terminal cannot be used continuously for long periods of time.

[0007] Therefore, a main object of the present invention is to provide a pedestrian device, a positioning system, and a positioning method that can perform visual positioning to obtain highly accurate position information, while suppressing power consumption and achieving power savings. [Means for solving the problem]

[0008] The pedestrian device of the present invention includes an external sensor that detects objects around the pedestrian, a memory that stores ambient environment information relating to the pedestrian's ambient environment, and a state sensor for detecting a state of a pedestrian; a processor, wherein the processor performs standard positioning to obtain position information of a pedestrian; Based on the detection result of the state sensor, when a predetermined change occurs in the state of a pedestrian, the specific event is determined as When a specific event occurs, visual positioning is performed to estimate the pedestrian's current location by comparing the image detected by the external sensor with the surrounding environment information, and the external sensor is turned on when the visual positioning is performed and turned off at other times.

[0010] The positioning system of the present invention is a positioning system configured with one or more computers that execute processing to acquire position information of a mobile body in a mobile body device, and includes an external sensor provided in the mobile body device that detects objects around the mobile body. a state sensor for detecting a state of a pedestrian; the computer has a memory that stores surrounding environment information related to the surrounding environment of the mobile object, and performs standard positioning to acquire position information of the mobile object; Based on the detection result of the state sensor, when a predetermined change occurs in the state of a pedestrian, the specific event is determined asWhen a specific event occurs, visual positioning is performed to estimate the current position of the moving body by comparing the image detected by the external sensor with the surrounding environment information, and the external sensor is turned on when the visual positioning is performed and turned off at other times.

[0011] Furthermore, a positioning method of the present invention is a positioning method in which a process of acquiring position information of a mobile body is executed by one or more computers in a mobile body device, the computer performing standard positioning to acquire the position information of the mobile body, Based on the detection result of a state sensor that detects the state of a pedestrian, when a predetermined change occurs in the state of a pedestrian, the specific event is When a specific event occurs, visual positioning is performed to estimate the current position of the mobile body by comparing the image detected by an external sensor that detects objects around the mobile body with environmental information about the mobile body's surrounding environment, and the external sensor is turned on when the visual positioning is performed and turned off at other times. [Effects of the Invention]

[0012] According to the present invention, it is possible to perform visual positioning to acquire highly accurate position information, and also to reduce power consumption and achieve power saving. [Brief explanation of the drawings]

[0013] [Figure 1] Overall configuration diagram of the traffic safety support system according to the first embodiment [Figure 2] FIG. 1 is an explanatory diagram showing an overview of visual positioning performed by a pedestrian terminal according to a first embodiment; [Figure 3] FIG. 1 is an explanatory diagram illustrating an overview of positioning control performed by a pedestrian terminal according to a first embodiment; [Figure 4] FIG. 1 is a block diagram showing a schematic configuration of a pedestrian terminal according to a first embodiment; [Figure 5] FIG. 1 is a block diagram showing a schematic configuration of a roadside device according to a first embodiment. [Figure 6] FIG. 1 is a flowchart showing an operation procedure of a pedestrian terminal according to a first embodiment; [Figure 7] FIG. 1 is a flowchart showing an operation procedure of a pedestrian terminal according to a first embodiment; [Figure 8] FIG. 1 is a flowchart showing an operation procedure of the in-vehicle terminal according to the first embodiment; [Figure 9] FIG. 1 is a flowchart showing an operation procedure of a roadside device according to a first embodiment; [Figure 10] FIG. 1 is a flowchart showing a procedure for visual positioning performed by a pedestrian terminal according to a first embodiment. [Figure 11] FIG. 10 is an explanatory diagram showing an overview of a foot image matching process performed by a pedestrian terminal according to a modification of the first embodiment; [Figure 12] FIG. 10 is a block diagram showing a schematic configuration of a pedestrian terminal according to a modification of the first embodiment; [Figure 13] FIG. 10 is an explanatory diagram illustrating an overview of positioning control performed by a pedestrian terminal according to a second embodiment. [Figure 14] FIG. 10 is a block diagram showing a schematic configuration of a pedestrian terminal according to a second embodiment. [Figure 15] FIG. 10 is a flowchart showing the operation procedure of the pedestrian terminal according to the second embodiment. [Figure 16] FIG. 11 is an explanatory diagram illustrating an overview of positioning control performed by a pedestrian terminal according to a third embodiment. [Figure 17] FIG. 10 is a block diagram showing a schematic configuration of a pedestrian terminal according to a third embodiment. [Figure 18] FIG. 10 is a flowchart showing the operation procedure of the pedestrian terminal according to the third embodiment. [Figure 19] FIG. 10 is an explanatory diagram illustrating an overview of positioning control performed by a pedestrian terminal according to a fourth embodiment. [Figure 20] FIG. 10 is an explanatory diagram showing an overview of image relative positioning performed by a pedestrian terminal according to a fourth embodiment; [Figure 21] FIG. 10 is a block diagram showing a schematic configuration of a pedestrian terminal according to a fourth embodiment. [Figure 22] FIG. 10 is a flowchart showing the operation procedure of the pedestrian terminal according to the fourth embodiment. [Figure 23] FIG. 13 is an explanatory diagram illustrating an overview of positioning control performed by a pedestrian terminal according to a fifth embodiment. [Figure 24] FIG. 10 is a block diagram showing a schematic configuration of a pedestrian terminal according to a fifth embodiment. [Figure 25] FIG. 10 is a flowchart showing the operation procedure of the pedestrian terminal according to the fifth embodiment. [Figure 26] FIG. 13 is an explanatory diagram illustrating an overview of positioning control performed by a pedestrian terminal according to a sixth embodiment. [Figure 27] FIG. 13 is a block diagram showing a schematic configuration of a pedestrian terminal according to a sixth embodiment. [Figure 28] FIG. 13 is a flowchart showing the operation procedure of the pedestrian terminal according to the sixth embodiment. [Figure 29] FIG. 20 is an explanatory diagram showing an overview of positioning control performed by a pedestrian terminal according to a seventh embodiment; [Figure 30] FIG. 13 is a block diagram showing a schematic configuration of a pedestrian terminal according to a seventh embodiment. [Figure 31] FIG. 13 is a flowchart showing the operation procedure of the pedestrian terminal according to the seventh embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] The first invention made to solve the above problem includes an external sensor that detects objects around a pedestrian, a memory that stores surrounding environment information about the pedestrian's surrounding environment, a state sensor for detecting a state of a pedestrian; a processor, wherein the processor performs standard positioning to obtain position information of a pedestrian; Based on the detection result of the state sensor, when a predetermined change occurs in the state of a pedestrian, the specific event is determined as When a specific event occurs, visual positioning is performed to estimate the pedestrian's current location by comparing the image detected by the external sensor with the surrounding environment information, and the external sensor is turned on when the visual positioning is performed and turned off at other times.

[0015] According to this, visual positioning is performed only in situations where high positioning accuracy is desired, so the frequency of visual positioning is reduced, thereby reducing power consumption and achieving energy savings. Furthermore, highly accurate visual positioning is performed in situations where there is a high possibility of a transition to a dangerous state for pedestrians, thereby enabling highly accurate risk assessment and improving pedestrian safety. Standard positioning is radio wave positioning that acquires location information based on incoming radio waves, such as satellite positioning that acquires location information based on radio waves from positioning satellites. Furthermore, visual positioning can be performed appropriately in situations where high positioning accuracy is required.

[0016] In a second aspect of the present invention, the external sensor is either a camera or a LIDAR.

[0017] This allows for accurate detection of objects around a pedestrian. Note that the external sensor is not limited to a camera or LIDAR. For example, the external sensor may be a radar.

[0020] Also, 3 In the invention, the status sensor is an acceleration sensor, and the processor is configured to perform the visual positioning when a predetermined change in the pedestrian's movement speed occurs as the specific event based on the detection result of the acceleration sensor.

[0021] According to this, if a pedestrian suddenly starts running or stops, it is assumed that the pedestrian has spotted an object that is dangerous to the pedestrian (such as a vehicle) or an object that the pedestrian is particularly interested in (an acquaintance), and there is a high possibility that the pedestrian will immediately take sudden action such as running out, which is a situation that requires high positioning accuracy. Therefore, by performing visual positioning at such times, pedestrian safety can be improved.

[0022] Also, 4 In the invention, the status sensor is a direction sensor, and the processor is configured to perform the visual positioning when a predetermined change in the direction of a pedestrian occurs as the specific event based on the detection result of the direction sensor.

[0023] According to this, for example, if a pedestrian suddenly changes their orientation (the direction of travel or the direction of their face), it is assumed that they have spotted an object that is dangerous to them (such as a vehicle) or an object that they are particularly interested in (an acquaintance), and there is a high possibility that they will immediately take sudden action such as running out into the street, which is a situation that requires high positioning accuracy. Therefore, by performing visual positioning at such times, pedestrian safety can be improved. Note that the orientation sensor is specifically a gyro sensor or a geomagnetic sensor.

[0024] Also, 5 The invention is equipped with a gaze sensor that detects the line of sight of a pedestrian, and the processor is configured to perform the visual positioning when a predetermined change in the line of sight of a pedestrian occurs as the specific event based on the detection result of the gaze sensor.

[0025] According to this, for example, if a pedestrian makes a large change in their line of sight, it is assumed that they have spotted an object that is dangerous to them (such as a vehicle) or an object that they are particularly interested in (an acquaintance), and there is a high possibility that they will immediately take sudden action such as running out into the street, which is a situation that requires high positioning accuracy. Therefore, by performing visual positioning at such times, pedestrian safety can be improved.

[0026] Also, 6 In the invention, the processor is configured to detect the specific event that requires the visual positioning to be performed based on the detection results of the plurality of status sensors.

[0027] This allows the detection results of multiple state sensors to be combined to detect signs of sudden behavior, such as a pedestrian suddenly running out, and appropriately determine the need to perform visual positioning.

[0028] Also, 7 The invention of The system includes an external sensor that detects objects around the pedestrian, a memory that stores ambient environment information relating to the pedestrian's ambient environment, and a processor, The processor , mark Satellite positioning is carried out as quasi-positioning. to obtain pedestrian location information, The trajectory of a pedestrian is acquired based on the position information acquired by the satellite positioning. ,before When a specific change occurs in the recorded trajectory When the specific event occurs, visual positioning is performed to estimate the current position of the pedestrian by comparing the image detected by the external sensor with the surrounding environment information, and the external sensor is turned on when the visual positioning is performed, and is turned off otherwise. The composition is as follows.

[0029] According to this, if an unusual change appears in a pedestrian's trajectory, that is, if the pedestrian itself behaves abnormally or if an abnormal trajectory appears due to a positioning error, visual positioning can be performed to obtain highly accurate location information, thereby improving pedestrian safety.

[0030] Also, 8 The invention of The system includes an external sensor that detects objects around the pedestrian, a memory that stores ambient environment information relating to the pedestrian's ambient environment, and a processor, The processor , mark Satellite positioning is carried out as quasi-positioning. to acquire pedestrian location information and When a specific defect occurs in the signal When the specific event occurs, visual positioning is performed to estimate the current position of the pedestrian by comparing the image detected by the external sensor with the surrounding environment information, and the external sensor is turned on when the visual positioning is performed, and is turned off otherwise. The composition is as follows.

[0031] This allows visual positioning to be performed appropriately when the accuracy of satellite positioning has decreased.

[0032] Also, 9 In the invention, the processor is configured to perform the visual positioning based on map information and pedestrian location information when the device enters an area with poor satellite reception registered in the map information.

[0033] This makes it possible to easily determine whether a specific defect appears in the satellite radio wave.

[0034] Also, 10 The invention of an external sensor that detects objects around the pedestrian; A foot camera that captures the road surface under pedestrians' feet a memory for storing surrounding environment information relating to the surrounding environment of a pedestrian; and a processor. wherein the processor: standard positioning is performed to acquire position information of the pedestrian, and when a specific event occurs, visual positioning is performed to estimate the current position of the pedestrian by comparing the image detected by the external sensor with the surrounding environment information, and control is performed to turn on the external sensor when the visual positioning is performed and turn off the external sensor at other times; said visual positioning So The system is configured to execute a first matching process that matches the detected image of the external sensor with the surrounding environment information, and a second matching process that matches the image captured by the foot camera with the surrounding environment information, and estimate the current position of the pedestrian based on the matching results of the first matching process and the second matching process.

[0035] This makes it possible to improve the positioning accuracy.

[0036] Also, 11 In the invention, even if a predetermined change does not occur in the pedestrian state, if the pedestrian's movement speed is faster than a predetermined value, the processor is configured to estimate the relative movement amount of the pedestrian from the previous position based on changes in the detected image of the external sensor at each time, and to perform image relative positioning at predetermined intervals to relatively estimate the pedestrian's current position.

[0037] According to this method, image relative positioning is performed when the pedestrian's movement speed is fast, and furthermore, image relative positioning is performed intermittently, which reduces power consumption. Note that image relative positioning has a smaller processing load and consumes less power than visual positioning (image absolute positioning) which involves image matching, but because it uses external sensors, power consumption increases if it is performed continuously.

[0038] Also, 12 In the invention, the processor is configured to perform the visual positioning when a predetermined change occurs in the pedestrian's state and the degree of change exceeds a predetermined range, and to perform image relative positioning to relatively estimate the pedestrian's current position by estimating the relative movement amount of the pedestrian from the previous position based on changes in the image detected by the external sensor at each time point, when a predetermined change occurs in the pedestrian's state and the degree of change does not exceed a predetermined range.

[0039] According to this, image relative positioning does not involve processing with a high load such as image matching, and therefore power consumption can be reduced.

[0040] Also, 13 In the invention, when a predetermined change does not occur in the pedestrian state, the processor estimates the relative movement amount of the pedestrian from the previous position based on the detection result of the state sensor, and performs non-image relative positioning to relatively estimate the current position of the pedestrian.

[0041] According to this, non-image relative positioning does not use external sensors and does not perform processing with a high load such as image matching, so power consumption can be reduced.

[0042] Also, 14In the invention, the processor is configured to perform the visual positioning when a predetermined change occurs in the pedestrian's state and the degree of change exceeds a predetermined range, and to perform route estimation positioning that determines the pedestrian's route pattern based on the pedestrian's most recent trajectory and estimates the pedestrian's current position when a predetermined change occurs in the pedestrian's state and the degree of change does not exceed a predetermined range.

[0043] According to this system, even if a predetermined change in the pedestrian's status occurs, if the degree of change is small, visual positioning is not performed and route estimation positioning is performed instead. This route estimation positioning does not use external sensors or perform heavy processing such as image matching, so power consumption can be reduced.

[0044] Also, 15 In the invention, the processor is configured to perform the visual positioning when the pedestrian approaches a predetermined safety area even if a predetermined change does not occur in the pedestrian state.

[0045] According to this, by performing visual positioning immediately before a pedestrian enters a safety area, it is possible to confirm that the pedestrian will definitely enter the safety area.

[0048] Also, 16 The invention is a positioning system that is configured with one or more computers that execute processing to acquire position information of a mobile body in a mobile body device, and includes an external sensor that is provided in the mobile body device and detects objects around the mobile body. a state sensor for detecting a state of a pedestrian; the computer has a memory that stores surrounding environment information related to the surrounding environment of the mobile object, and performs standard positioning to acquire position information of the mobile object; Based on the detection result of the state sensor, when a predetermined change occurs in the state of a pedestrian, the specific event is determined as When a specific event occurs, visual positioning is performed to estimate the current position of the moving body by comparing the image detected by the external sensor with the surrounding environment information, and the external sensor is turned on when the visual positioning is performed and turned off at other times.

[0049] According to this, as in the first invention, highly accurate position information can be obtained by performing visual positioning, and power consumption can be reduced to achieve power saving. Furthermore, visual positioning can be performed appropriately in situations where high positioning accuracy is required.

[0050] Also, 17 The present invention is a positioning method in which one or more computers execute a process for acquiring position information of a mobile body in a mobile body device, the computer performing standard positioning to acquire the position information of the mobile body, Based on the detection result of a state sensor that detects the state of a pedestrian, when a predetermined change occurs in the state of a pedestrian, the specific event is When a specific event occurs, visual positioning is performed to estimate the current position of the mobile body by comparing the image detected by an external sensor that detects objects around the mobile body with environmental information about the mobile body's surrounding environment, and the external sensor is turned on when the visual positioning is performed and turned off at other times.

[0051] According to this, as in the first invention, highly accurate position information can be obtained by performing visual positioning, and power consumption can be reduced to achieve power saving. Furthermore, visual positioning can be performed appropriately in situations where high positioning accuracy is required.

[0052] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0053] (First embodiment) FIG. 1 is a diagram showing the overall configuration of a traffic safety support system according to the first embodiment.

[0054] This traffic safety support system supports traffic safety for pedestrians and vehicles, and includes a pedestrian terminal 1 (pedestrian device, mobile device, computer), an in-vehicle terminal 2 (in-vehicle device), and a roadside unit 3 (roadside device).

[0055] ITS communication is performed between the pedestrian terminal 1, the in-vehicle terminal 2, and the roadside unit 3. This ITS communication is wireless communication using a frequency band (e.g., 700 MHz band or 5.8 GHz band) adopted in a safe driving support wireless system that uses an ITS (Intelligent Transport System). In this embodiment, ITS communication between the pedestrian terminal 1 and the in-vehicle terminal 2 will be referred to as pedestrian-to-vehicle communication, ITS communication between the pedestrian terminal 1 and the roadside unit 3 will be referred to as road-to-pedestrian communication, and ITS communication between the in-vehicle terminal 2 and the roadside unit 3 will be referred to as road-to-vehicle communication. ITS communication is also performed between in-vehicle terminals 2, and this ITS communication will be referred to as vehicle-to-vehicle communication.

[0056] Pedestrian terminal 1 is carried by a pedestrian. This pedestrian terminal 1 sends and receives messages including location information etc. to and from in-vehicle terminal 2 via ITS communication (pedestrian-to-vehicle communication), determines the risk of collision between the pedestrian and a vehicle, and if there is a risk of collision, activates an action to warn the pedestrian. Pedestrian terminal 1 is a wearable device attached to the body of a pedestrian, particularly a wearable device attached to the head of a pedestrian, a device known as a head-mounted device or smart glasses, and is equipped with a function to realize AR (Augmented Reality).

[0057] The in-vehicle terminal 2 is mounted on a vehicle. This in-vehicle terminal 2 transmits and receives messages including location information and the like to and from the pedestrian terminal 1 via ITS communication (pedestrian-to-vehicle communication), determines the risk of collision between the pedestrian and the vehicle, and performs an attention-calling operation for the driver if there is a risk of collision. The attention-calling operation may be performed using, for example, a car navigation device.

[0058] The roadside unit 3 is installed at road intersections, etc. The roadside unit 3 distributes various information such as traffic information to the pedestrian terminal 1 and the in-vehicle terminal 2 through ITS communication (road-to-pedestrian communication, road-to-vehicle communication). The roadside unit 3 also notifies the in-vehicle terminal 2 and the pedestrian terminal 1 of the presence of vehicles and pedestrians located around the unit through ITS communication (road-to-vehicle communication, road-to-pedestrian communication). This makes it possible to prevent collisions at intersections where there is no line of sight.

[0059] Here, the pedestrian terminal 1 constantly performs satellite positioning as standard positioning to acquire position information of the pedestrian's current location. Satellite positioning acquires position information based on radio waves from positioning satellites. Note that standard positioning may also be radio wave positioning that acquires position information based on incoming radio waves, for example, positioning that acquires position information based on radio waves from a terrestrial base station. Standard positioning may also be positioning using PDR (pedestrian dead-reckoning). In this PDR positioning, the pedestrian terminal 1 estimates the relative movement amount of the pedestrian from the previous position based on detection results from a gyro sensor, acceleration sensor, etc., and relatively estimates the pedestrian's current position.

[0060] Next, the visual positioning performed by the pedestrian terminal 1 according to the first embodiment will be described. FIG.

[0061] The pedestrian terminal 1 performs visual positioning to acquire location information of the pedestrian's current location using images captured by the camera 12 that captures the pedestrian's surroundings. In visual positioning, the captured images output in real time from the camera 12 (hereinafter referred to as "real-time captured images") are compared with candidate images registered in a surrounding environment DB (surrounding environment database), and the location information corresponding to the candidate image that has been successfully matched is acquired as the location information of the pedestrian's current location.

[0062] The surrounding environment DB stores information about the surrounding environment of a pedestrian (surrounding environment information). Specifically, an image captured on a road where a pedestrian is walking, in a direction that corresponds to the line of sight of the pedestrian looking ahead, is associated with the location information of the capture point and registered in the surrounding environment DB. Note that instead of the captured image itself, feature information (information on feature points) extracted from the captured image may also be registered in the surrounding environment DB.

[0063] The surrounding environment DB may also store three-dimensional map information (environmental map information). The three-dimensional map information is three-dimensional information about objects present in the scenery visible to pedestrians, specifically, roads and fixed structures such as buildings in the vicinity. This three-dimensional map information is generated based on, for example, images captured from various directions of a target space including roads on which pedestrians pass.

[0064] The roadside device 3 stores registration information in a surrounding environment DB relating to a predetermined area around the roadside device 3. When the roadside device 3 approaches the pedestrian terminal 1, the roadside device 3 distributes the registration information in the surrounding environment DB of the roadside device 3 to the pedestrian terminal 1.

[0065] Next, a description will be given of the positioning control performed by the pedestrian terminal 1 according to the first embodiment. FIG.

[0066] With satellite positioning, the positioning accuracy decreases due to the effects of multipath and the blockage of satellite radio waves caused by pedestrians entering the shadows of buildings. On the other hand, visual positioning can ensure high accuracy, but it consumes a lot of power because it requires operating a camera and the load of matching processing is heavy. For this reason, if visual positioning is performed frequently, there is a problem that the pedestrian terminal 1 cannot be used continuously for long periods of time. For this reason, it is desirable to reduce power consumption and achieve power savings.

[0067] Therefore, in this embodiment, the pedestrian terminal 1 constantly performs satellite positioning, and performs visual positioning only when a specific event occurs that requires high positioning accuracy. As a result, visual positioning is performed only in situations where high positioning accuracy is desired, reducing the frequency of visual positioning, thereby reducing power consumption and achieving energy savings. Furthermore, by performing highly accurate visual positioning in situations where there is a high possibility of a transition to a dangerous state for pedestrians, highly accurate risk judgment is achieved, thereby improving pedestrian safety.

[0068] In particular, in this embodiment, visual positioning is performed when a predetermined change occurs in the pedestrian state in the pedestrian terminal 1. Specifically, the moving speed, orientation (direction of travel and face direction), and line of sight are detected as the pedestrian state, and visual positioning is performed when a predetermined change occurs in any of these.

[0069] For example, if a pedestrian suddenly starts running or suddenly stops, suddenly changes the direction of travel or the direction of their face, or makes a large change in their line of sight, it is assumed that the pedestrian has spotted an object that is dangerous to the pedestrian (such as a vehicle) or an object that the pedestrian is particularly interested in (an acquaintance). When a pedestrian behaves in this way, there is a high possibility that the pedestrian will immediately take a sudden action, such as jumping out, creating a dangerous situation, so high positioning accuracy is required. Therefore, visual positioning at such times can increase pedestrian safety.

[0070] In the example shown in Figure 3(A), the pedestrian suddenly starts running, so it is determined that a predetermined change has occurred in the pedestrian's state, and visual positioning is performed. On the other hand, in the example shown in Figure 3(B), the pedestrian is walking at a roughly constant speed, so it is determined that no predetermined change has occurred in the pedestrian's state, and visual positioning is not performed.

[0071] Next, a description will be given of the schematic configuration of the pedestrian terminal 1 according to the first embodiment. Fig. 4 is a block diagram showing the schematic configuration of the pedestrian terminal 1.

[0072] The pedestrian terminal 1 includes a satellite positioning unit 11, a camera 12 (external environment sensor), a status sensor 13, an AR display 14, an ITS communication unit 15, a wireless communication unit 16, a memory 17, and a processor 18.

[0073] Satellite positioning unit 11 measures the position of the device itself using a satellite positioning system such as GPS (Global Positioning System) or QZSS (Quasi-Zenith Satellite System), and acquires position information (latitude and longitude) of the device itself.

[0074] The camera 12 captures an image of the area in front of the pedestrian.

[0075] The state sensor 13 detects the state of the pedestrian. In this embodiment, the pedestrian terminal 1 is equipped with an acceleration sensor 21, a gyro sensor 22 (angular velocity sensor), a geomagnetic sensor 23, and a gaze sensor 24 as the state sensor 13. The acceleration sensor 21 detects the acceleration occurring in the body of the pedestrian. The gyro sensor 22 detects the angular velocity occurring in the body of the pedestrian. The geomagnetic sensor 23 detects the geomagnetic direction. The gaze sensor 24 (gaze camera) captures the left and right eyes of the pedestrian.

[0076] The AR display 14 superimposes and displays a virtual object on a real space within the pedestrian's field of vision, thereby realizing AR (Augmented Reality).

[0077] The ITS communication unit 15 broadcasts messages to the in-vehicle terminal 2 and the roadside unit 3 through ITS communication (pedestrian-to-vehicle communication and road-to-pedestrian communication), and also receives messages transmitted from the in-vehicle terminal 2 and the roadside unit 3.

[0078] The wireless communication unit 16 transmits messages to the roadside unit 3 and receives messages transmitted from the roadside unit 3 by wireless communication such as WiFi (registered trademark).

[0079] The memory 17 stores map information, programs executed by the processor 18, and the like. The memory 17 also stores registration information (candidate images and location information) in the surrounding environment DB. In this embodiment, when the pedestrian terminal 1 approaches an intersection, it acquires registration information in the surrounding environment DB related to the area surrounding the intersection from the roadside unit 3 installed at the intersection. In addition, feature information (information on a plurality of feature points) extracted from the candidate image may be registered in the surrounding environment DB, rather than the candidate image itself.

[0080] The processor 18 performs various processes by executing programs stored in the memory 17. In this embodiment, the processor 18 performs a message control process P1, a collision determination process P2, an attention-calling control process P3, a state information acquisition process P4, a pedestrian state determination process P5, a positioning control process P6, a camera control process P7, and a visual positioning process P8.

[0081] In the message control process P1, the processor 18 controls the transmission and reception of messages of ITS communication between the in-vehicle terminal 2 and the roadside device 3. The processor 18 also controls the transmission and reception of messages of wireless communication with the roadside device 3.

[0082] In the collision determination process P2, the processor 18 determines whether there is a risk of the vehicle colliding with a pedestrian based on the vehicle position information contained in the vehicle information obtained from the in-vehicle terminal 2 and the pedestrian position information obtained by the satellite positioning unit 11.

[0083] In the attention-calling control process P3, the processor 18 performs control so as to perform a predetermined attention-calling action (such as audio output or vibration) for the pedestrian when it is determined in the collision determination process P2 that there is a risk of collision.

[0084] In the state information acquisition process P4, the processor 18 acquires state information representing the state of the pedestrian based on the detection result of the state sensor 13.

[0085] Specifically, the processor 18 measures the moving speed of the pedestrian based on the detection result of the acceleration sensor 21. When the pedestrian walks, acceleration occurs in the body of the pedestrian, and the walking pitch of the pedestrian is calculated based on the change in this acceleration. Furthermore, the speed is calculated from the walking pitch and stride length. The stride length may be set based on the attributes of the pedestrian (adult, child, etc.) registered in the pedestrian terminal 1.

[0086] Furthermore, based on the detection results of the gyro sensor 22 and the geomagnetic sensor 23, the processor 18 measures the head orientation (face direction) of the pedestrian wearing the pedestrian terminal 1 and the moving direction of the pedestrian.

[0087] Furthermore, the processor 18 measures the line of sight (viewpoint) of the pedestrian (line of sight sensing) based on the detection result (captured image) of the line of sight sensor 24. Specifically, the processor 18 acquires position information of the pedestrian's line of sight, i.e., the coordinate value of the line of sight in the coordinate system of the pedestrian's field of view.

[0088] In the pedestrian state determination process P5, the processor 18 determines whether a predetermined change has occurred in the pedestrian state based on the state information acquired in the state information acquisition process P4. At this time, the amount of change in the pedestrian state from the standard state and the rate of change in the pedestrian state (amount of change per unit time) are compared with predetermined thresholds to determine whether a predetermined change has occurred in the pedestrian state. That is, if the pedestrian state changes significantly and the amount of change from the standard state is equal to or greater than the predetermined threshold, or if the pedestrian state changes suddenly and the rate of change is equal to or greater than the predetermined threshold, it is determined that a predetermined change has occurred in the pedestrian state.

[0089] Specifically, when a pedestrian starts running, that is, when the amount of change from the standard walking speed exceeds a predetermined threshold, it is determined that a predetermined change has occurred in the pedestrian's state. Also, when a pedestrian suddenly starts running (sudden acceleration) or suddenly stops (sudden stop), that is, when the rate of change in speed exceeds a predetermined threshold, it is determined that a predetermined change has occurred in the pedestrian's state.

[0090] Regarding the pedestrian's orientation (direction of travel or direction of face), for example, if the pedestrian significantly changes their direction of travel or direction of face, i.e., if the amount of change in orientation from a standard state facing forward exceeds a predetermined threshold, it is determined that a predetermined change has occurred in the pedestrian's state. Also, for example, if the pedestrian suddenly changes their direction of travel or direction of face, i.e., if the rate of change in orientation exceeds a predetermined threshold, it is determined that a predetermined change has occurred in the pedestrian's state.

[0091] Furthermore, regarding the gaze direction (viewpoint) of a pedestrian, for example, if the gaze direction of the pedestrian changes significantly, i.e., if the amount of change in the gaze direction from a standard state facing forward becomes equal to or greater than a predetermined threshold, it is determined that a predetermined change has occurred in the pedestrian's state. Furthermore, for example, if the gaze direction of the pedestrian changes suddenly, i.e., if the rate of change in the gaze direction becomes equal to or greater than a predetermined threshold, it is determined that a predetermined change has occurred in the pedestrian's state.

[0092] In the positioning control process P6, the processor 18 instructs the execution of a visual positioning process P8 when a predetermined change in the pedestrian state occurs as a result of the determination in the pedestrian state determination process P5.

[0093] In camera control processing P7, processor 18 controls the on / off of camera 12 depending on the timing of visual positioning, which requires camera 12. That is, camera 12 is turned on when visual positioning is performed, and is turned off when visual positioning is not performed. Note that the on / off control of camera 12 controls the power supply to components of camera 12, such as the imaging element and signal processing circuit, but it may also control the power supply to some of the components of camera 12.

[0094] In visual positioning process P8, processor 18 compares the captured image output in real time from camera 12 with candidate images extracted from the surrounding environment DB of the host device to estimate the current position of the pedestrian. This visual positioning process P8 includes a comparison target extraction process P9, an image comparison process P10, and a position information acquisition process P11.

[0095] In the matching target extraction process P9, the processor 18 extracts candidate images to be matched from the surrounding environment DB of the own device.

[0096] In the image matching process P10, the processor 18 matches the captured image output in real time from the camera 12 with the candidate image extracted in the matching target extraction process P9. At this time, the processor 18 extracts feature information (information on a plurality of feature points) from each of the captured image and the candidate image, and compares the feature information to perform image matching.

[0097] In the position information acquisition process P11, the processor 18 acquires the position information associated with the candidate image that has been successfully matched in the image matching process P10 as the position information of the pedestrian's current location.

[0098] The in-vehicle terminal 2 also includes a processor and a memory (not shown), and can perform message control processing, collision determination processing, and attention-call control processing by executing programs stored in the memory.

[0099] Furthermore, in this embodiment, the pedestrian terminal 1 is configured to perform visual positioning when a specific event requiring high positioning accuracy occurs, i.e., when a predetermined change appears in the pedestrian's state, but a similar configuration may be performed by the in-vehicle terminal 2. That is, the in-vehicle terminal 2 may be configured to perform visual positioning when a specific event occurs, for example, when a predetermined change appears in the vehicle state.

[0100] In this embodiment, the pedestrian terminal 1 includes a camera 12 as an external sensor that detects objects around the pedestrian. However, the external sensor is not limited to the camera 12. For example, the external sensor may be a LIDAR (Light Detection and Ranging) sensor that detects objects using laser light. Alternatively, the external sensor may be a radar that detects objects using radio waves. Alternatively, the external sensor may be a sensor having a depth prediction function (a depth measurement function) that measures depth, i.e., the distance (depth) from the external sensor to an object. In visual positioning, a detected image output in real time from such an external sensor is compared with a candidate image registered in the surrounding environment DB. Note that the image in this embodiment may include not only two-dimensional information but also three-dimensional information, such as a distance image acquired by a depth camera. Furthermore, the image matching process P10 may be performed by matching two pieces of two-dimensional information together, or by matching two pieces of three-dimensional information together, or by matching two pieces of two-dimensional information together with three-dimensional information.

[0101] In addition, in this embodiment, the camera 12 is configured to capture an image in front of the pedestrian, but the camera 12 may be configured to capture an image in any direction around the pedestrian, for example, behind the pedestrian, or to capture an image in a wider range around the pedestrian.

[0102] Next, a description will be given of the general configuration of the roadside unit 3. FIG.

[0103] The roadside unit 3 includes an ITS communication unit 31, a wireless communication unit 32, a memory 33, and a processor .

[0104] The ITS communication unit 31 broadcasts messages to the pedestrian terminal 1 and the in-vehicle terminal 2 through ITS communication (road-to-pedestrian communication, road-to-vehicle communication), and also receives messages transmitted from the pedestrian terminal 1 and the in-vehicle terminal 2.

[0105] The wireless communication unit 32 transmits messages to the pedestrian terminal 1 and receives messages transmitted from the pedestrian terminal 1 via wireless communication such as WiFi (registered trademark).

[0106] The memory 33 stores programs executed by the processor 34. In this embodiment, the memory 33 also stores registration information of the surrounding environment DB.

[0107] The processor 34 performs various processes by executing programs stored in the memory 33. In this embodiment, the processor 34 performs message control processing and surrounding environment DB management processing.

[0108] In the message control process, the processor 34 controls the transmission and reception of messages of ITS communication between the pedestrian terminal 1 and the in-vehicle terminal 2. The processor 34 also controls the transmission and reception of messages of wireless communication with the pedestrian terminal 1.

[0109] In the surrounding environment DB management process, the processor 34 manages the surrounding environment DB. Specifically, in response to a request from the pedestrian terminal 1, the processor 34 distributes registration information of the surrounding environment DB from the wireless communication unit 32 to the pedestrian terminal 1.

[0110] The functions of the roadside device 3 may be provided in a cloud computer. For example, in the present embodiment, the surrounding environment DB is provided in the roadside device 3, but the surrounding environment DB may also be provided in a cloud computer. In this case, the pedestrian terminal 1 may be configured to download the registration information of the surrounding environment DB by communicating with the cloud computer via the roadside device 3. Furthermore, the pedestrian terminal 1 may be configured to have a high-speed cellular communication function such as 5G and to download the registration information of the surrounding environment DB by communicating with the cloud computer via a cellular communication network or the like. In this case, the pedestrian terminal 1 may be configured to notify the cloud computer of its own location information, so that the cloud computer may distribute the registration information of the surrounding environment DB for a required range around the pedestrian terminal 1 to the pedestrian terminal 1.

[0111] Furthermore, the cloud computer may be provided with the functions of the pedestrian terminal 1. For example, in the first embodiment, the pedestrian terminal 1 performs processes such as pedestrian state determination and visual positioning, but these processes may be performed by the cloud computer.

[0112] Next, the operation procedures of the pedestrian terminal 1, the in-vehicle terminal 2, and the roadside device 3 according to the first embodiment will be described. FIGS. 6 and 7 are flow diagrams showing the operation procedures of the pedestrian terminal 1. FIG. 8 is a flow diagram showing the operation procedures of the in-vehicle terminal 2. FIG. 9 is a flow diagram showing the operation procedures of the roadside device 3. Note that each process shown in FIGS. 6(A), (B), (C), FIG. 8, and FIGS. 9(A) and (B) is performed at predetermined intervals. In other words, these processes are repeatedly executed even after they have been completed.

[0113] 6(A), in the pedestrian terminal 1, first, the satellite positioning unit 11 acquires the position information of the pedestrian (ST101). Next, the processor 18 determines, based on the position information of the pedestrian, whether or not the situation requires the pedestrian information to be transmitted, specifically, whether or not the pedestrian has entered a dangerous area (for example, an intersection) (ST102).

[0114] Here, if the situation requires the pedestrian information to be transmitted (Yes in ST102), the ITS communication unit 15 transmits an ITS communication message including the pedestrian information (such as the pedestrian ID and location information) to the in-vehicle terminal 2 and the roadside unit 3 in response to a transmission instruction from the processor 18 (ST103).

[0115] As shown in FIG. 8, when the in-vehicle terminal 2 receives an ITS communication (pedestrian-to-vehicle communication) message from the pedestrian terminal 1 (Yes in ST201), it performs a collision determination to determine whether there is a risk of the vehicle colliding with a pedestrian based on the vehicle position information contained in the message (ST202).

[0116] If there is a risk of the host vehicle colliding with a pedestrian (Yes in ST202), a predetermined attention-calling operation is performed for the driver (ST203). Specifically, as the attention-calling operation, the car navigation device is made to perform an attention-calling operation (for example, audio output or screen display). If the host vehicle is an autonomous vehicle, the autonomous driving ECU (cruise control device) is instructed to perform a predetermined collision avoidance operation.

[0117] 9(A), in the roadside device 3, when the ITS communication unit 31 receives a message of ITS communication (pedestrian-to-vehicle communication) from the pedestrian terminal 1 (Yes in ST301), the processor 34 acquires the terminal ID and location information of the pedestrian terminal 1 included in the received message (ST302). Next, based on the pedestrian's location information, the processor 34 determines whether the pedestrian terminal 1 is located in the vicinity of (inside or near) the target area of ​​the registered information in the surrounding environment DB (ST303).

[0118] Here, if the pedestrian terminal 1 is located in the vicinity of the target area (Yes in ST303), in response to a transmission instruction from the processor 34, the ITS communication unit 31 transmits to the pedestrian terminal 1 an ITS communication message including DB usage information indicating that the pedestrian terminal 1 can use the registered information in the surrounding environment DB of the device (ST304).

[0119] As shown in FIG. 6(B), in the pedestrian terminal 1, when the ITS communication unit 15 receives an ITS communication message including DB usage information from the roadside unit 3 (Yes in ST111), in response to a transmission instruction from the processor 18, the wireless communication unit 16 transmits a wireless communication message to the roadside unit 3 requesting DB registration information (registration information of the surrounding environment DB) (ST112).

[0120] As shown in Figure 9 (B), in the roadside unit 3, when the wireless communication unit 32 receives a wireless communication message requesting DB registration information from the pedestrian terminal 1 (Yes in ST311), in response to a transmission instruction from the processor 34, the wireless communication unit 32 transmits a wireless communication message including the DB registration information (registration information of the surrounding environment DB) to the pedestrian terminal 1 (ST312).

[0121] At this time, all of the registered information in the surrounding environment DB of the roadside device 3 may be transmitted to the pedestrian terminal 1, or only a portion of the registered information that is likely to be used by the pedestrian terminal 1 may be transmitted to the pedestrian terminal 1. Specifically, the registered information within a predetermined range around the pedestrian terminal 1, particularly within a predetermined range located in the direction of travel of the pedestrian, may be transmitted to the pedestrian terminal 1.

[0122] As shown in Figure 6 (C), in the pedestrian terminal 1, when the wireless communication unit 16 receives a wireless communication message including DB registration information from the roadside unit 3 (Yes in ST121), the processor 18 registers the DB registration information (registration information of the surrounding environment DB) included in the received message in the surrounding environment DB of the device itself (ST122).

[0123] Next, as shown in FIG. 7, in the pedestrian terminal 1, the processor 18 acquires the position information of the pedestrian from the satellite positioning unit 11 (ST131).

[0124] Next, processor 18 acquires state information of the pedestrian based on the detection result of state sensor 13 (ST132). Specifically, processor 18 measures the moving speed of the pedestrian based on the detection result of acceleration sensor 21. Processor 18 also measures the moving direction (face direction) of the pedestrian based on the detection results of gyro sensor 22 and geomagnetic sensor 23. Processor 18 also measures the line of sight of the pedestrian based on the detection result of line of sight sensor 24.

[0125] Next, processor 18 determines whether or not a predetermined change has occurred in the state of the pedestrian based on the state information of the pedestrian (ST133).

[0126] If a predetermined change in the pedestrian status occurs (Yes in ST133), then processor 18 turns on camera 12 (ST134). At this time, processor 18 starts camera 12 if it is off, and does not perform any particular control if camera 12 is already on. Next, processor 18 performs visual positioning (ST135).

[0127] On the other hand, if the predetermined change in the pedestrian state does not occur (No in ST133), processor 18 turns off camera 12 (ST136). At this time, processor 18 stops camera 12 if it is in the on state, and does not perform any particular control if camera 12 is already in the off state. In this case, the positioning result of the satellite positioning is used as the position information of the pedestrian.

[0128] Next, the procedure of visual positioning (ST135 in FIG. 7) performed by the pedestrian terminal 1 according to the first embodiment will be described. FIG.

[0129] In the pedestrian terminal 1, first, the processor 18 acquires a real-time captured image from the camera 12 (ST401).

[0130] Next, processor 18 extracts candidate images to be matched from the surrounding environment DB of the own device as a match target extraction process P9 (ST402).

[0131] Next, processor 18 performs image matching processing P10 by matching candidate images extracted from the surrounding environment DB of the own device with real-time captured images (ST403).

[0132] If this image matching is successful, i.e., if the candidate image and the captured image match (Yes in ST404), processor 18 acquires the location information associated with the successfully matched candidate image as the location information of the pedestrian's current location (ST405) as location information acquisition process P11.

[0133] Incidentally, in the pedestrian state determination process P5, the processor 18 may combine multiple state information (movement speed, direction, line of sight direction) based on the detection results of the multiple state sensors 13 to detect a specific event that requires visual positioning.

[0134] For example, it is possible to detect, for example, a state in which a pedestrian suddenly stops and turns around by combining the pedestrian's movement speed based on the detection result of the acceleration sensor 21 with the pedestrian's facial direction based on the detection result of the gyro sensor 22 or the geomagnetic sensor 23. Furthermore, the processor 18 may combine the pedestrian's movement speed based on the detection result of the acceleration sensor 21 with the pedestrian's gaze direction based on the detection result of the gaze sensor 24 to detect, for example, a state in which a pedestrian makes a large or sudden change in their gaze and then starts running as a sign of dangerous pedestrian behavior.

[0135] In such cases, there is a high possibility that the pedestrian will immediately take sudden action, such as jumping out into the street, and by detecting such a situation as a sign of dangerous pedestrian behavior, it is possible to appropriately determine the need to perform visual positioning.

[0136] (Modification of the first embodiment) Next, a modified example of the first embodiment will be described. Note that points not particularly mentioned here are the same as those of the above-described embodiment. Fig. 11 is an explanatory diagram showing an overview of the foot image matching process performed by a pedestrian terminal 1 according to a modified example of the first embodiment. Fig. 12 is a block diagram showing the schematic configuration of a pedestrian terminal 1 according to a modified example of the first embodiment.

[0137] In the first embodiment, visual positioning is performed using an image captured by camera 12 that captures an image in front of the pedestrian. On the other hand, in this modified example, visual positioning is performed using a front image that captures an image in front of the pedestrian and a foot image that captures an image of the pedestrian's feet.

[0138] In this modification, the pedestrian terminal 1 is equipped with a front camera 25 that captures an image in front of the pedestrian and a foot camera 26 that captures an image of the pedestrian's feet (see FIG. 12). Note that one camera may be configured to have a wide angle of view so that it can capture an image of a wide range including an image in front of and under the pedestrian's feet.

[0139] Road surfaces gradually deteriorate over time. For example, road surfaces are painted with white lines and other road markings using special traffic paint. These road markings can develop cracks and other deterioration. Asphalt pavement can also develop defects and other deterioration. The state of road surface deterioration has unique characteristics for each location. Therefore, the location where the underfoot image was captured can be identified based on the road surface characteristics.

[0140] Therefore, in this modification, a photographed image of the road surface at a registered point is registered in the surrounding environment DB in association with the position information of the registered point for use in foot image matching. Also, as in the first embodiment, a photographed image taken in an imaging direction corresponding to the line of sight of the pedestrian looking ahead is registered in the surrounding environment DB in association with the position information of the registered point for use in front image matching.

[0141] Meanwhile, in the pedestrian terminal 1, the front camera 25 outputs a front image captured in front of the pedestrian in real time. As a visual positioning process P8, the processor 18 extracts candidate images for front image matching (images captured at registered points) registered in the surrounding environment DB (front matching target extraction process P12), and matches the candidate images for front image matching with the front image output in real time from the front camera 25 (front image matching process P13, first matching process).

[0142] Furthermore, in the pedestrian terminal 1, the foot camera 26 outputs in real time foot images of the road surface under the pedestrian's feet. As a visual positioning process P8, the processor 18 extracts candidate images for foot image matching (images taken at registered points) registered in the surrounding environment DB (foot matching target extraction process P14), and matches the candidate images for foot image matching with the foot images output in real time from the foot camera 26 (foot image matching process P15, second matching process).

[0143] Next, processor 18 acquires position information of the pedestrian's current location based on position information corresponding to the candidate image successfully matched in the forward image matching process P13 and position information corresponding to the candidate image successfully matched in the feet image matching process P15 (position information acquisition process P11). At this time, the position information of the pedestrian's current location may be acquired by performing appropriate statistical processing (averaging) on ​​the position information based on the forward image matching process P13 and the position information based on the feet image matching process P15.

[0144] Furthermore, because the position information based on the foot image matching process P15 is more accurate than the position information based on the forward image matching process P13, the forward image matching process P13 may be used as a provisional positioning, and the foot image matching process P15 may be used as a final positioning, and the candidate images in the foot image matching process P15 may be narrowed down using the matching results of the forward image matching process P13. Specifically, the pedestrian's position based on the forward image matching process P13 is used as the pedestrian's provisional position, and candidate images for foot image matching within a predetermined range around the provisional position are extracted from the surrounding environment DB, and the foot image matching process P15 is then performed. This reduces the processing load of image matching.

[0145] (Second embodiment) Next, a second embodiment will be described. Note that points not particularly mentioned here are the same as those in the above-described embodiment. Fig. 13 is an explanatory diagram showing an overview of positioning control performed by a pedestrian terminal 1 according to the second embodiment.

[0146] In the first embodiment, the pedestrian terminal 1 performs visual positioning when a predetermined change appears in the pedestrian's state based on state information acquired by the state sensor 13. On the other hand, in the present embodiment, the pedestrian terminal 1 performs visual positioning when a predetermined change appears in the pedestrian's trajectory based on position information acquired by satellite positioning.

[0147] If a pedestrian behaves abnormally due to poor health or other reasons, there is a high risk of the pedestrian colliding with a vehicle, so high positioning accuracy is required. Furthermore, with satellite positioning, if a pedestrian enters the shadow of a building, blocking satellite radio waves, or the effects of multipath can reduce positioning accuracy, making it impossible to properly determine a collision due to positioning errors, so high-precision positioning is required.

[0148] On the other hand, if the pedestrian himself / herself behaves abnormally (see FIG. 13(A)), or if the accuracy of satellite positioning decreases and a large positioning error appears (see FIG. 13(B)), an abnormal change will appear in the pedestrian's trajectory. Therefore, in this embodiment, if an abnormal change appears in the pedestrian's trajectory, visual positioning is performed to obtain highly accurate position information, thereby improving the safety of the pedestrian.

[0149] Next, a description will be given of the schematic configuration of the pedestrian terminal 1 according to the second embodiment. Fig. 14 is a block diagram showing the schematic configuration of the pedestrian terminal 1. The schematic configuration of the roadside device 3 is the same as that of the first embodiment (see Fig. 5).

[0150] In this embodiment, the processor 18 of the pedestrian terminal 1 performs the trajectory determination process P21. The rest is substantially the same as in the first embodiment (see FIG. 4).

[0151] In the trajectory determination process P21, the processor 18 determines whether a predetermined change has occurred in the trajectory of the pedestrian based on the position information of the pedestrian at each time. At this time, for example, if the position of the pedestrian determined by satellite positioning shows a large or sudden change that would not occur under normal conditions, it is determined that a predetermined change has occurred in the pedestrian's trajectory.

[0152] In the positioning control process P6, the processor 18 instructs the execution of a visual positioning process P8 when a predetermined change appears in the pedestrian's trajectory as a result of the determination in the trajectory determination process P21.

[0153] In this embodiment, the status sensor 13 provided in the pedestrian terminal 1 in the first embodiment is omitted, but the pedestrian terminal 1 may be configured to include the status sensor 13 and perform both the pedestrian status determination process P5 and the trajectory determination process P21.

[0154] Next, the operation procedure of the pedestrian terminal 1 according to the second embodiment will be described. Fig. 15 is a flow diagram showing the operation procedure of the pedestrian terminal 1. Note that the pedestrian terminal 1 performs the same processing as in the first embodiment (see Figs. 6(A), (B), and (C)). Also, the operation procedure of the roadside device 3 is the same as in the first embodiment (see Fig. 9).

[0155] In the pedestrian terminal 1, the processor 18 acquires the position information of the pedestrian from the satellite positioning unit 11 (ST131). Next, the processor 18 determines whether or not a predetermined change has occurred in the trajectory of the pedestrian based on the position information of the pedestrian at each time (ST141).

[0156] Here, if a predetermined change appears in the trajectory of the pedestrian (Yes in ST141), the processor 18 turns on the camera 12 (ST134) and performs visual positioning (ST135).

[0157] On the other hand, if the predetermined change does not appear in the trajectory of the pedestrian (No in ST141), the processor 18 turns off the camera 12 (ST136). In this case, the positioning result of the satellite positioning is used as the position information of the pedestrian.

[0158] (Third embodiment) Next, a third embodiment will be described. Note that the points not specifically mentioned here are the same as those in the previous embodiments. Fig. 16 is an explanatory diagram showing an overview of positioning control performed by a pedestrian terminal 1 according to the third embodiment.

[0159] In the first embodiment, the pedestrian terminal 1 performs visual positioning when a predetermined change occurs in the pedestrian's state based on the detection result of the state sensor 13. On the other hand, in the present embodiment, the pedestrian terminal 1 performs visual positioning when a predetermined defect occurs in the satellite radio waves.

[0160] Specifically, when a pedestrian enters an area with poor satellite signal reception, such as an area with high-rise buildings (see FIG. 16(A)), the pedestrian terminal 1 determines that the situation is such that a predetermined defect occurs in the satellite signal, and performs visual positioning. Also, when the pedestrian terminal 1 cannot observe the required number of positioning satellites (the number that can ensure the required accuracy) (see FIG. 16(B)), it determines that the situation is such that a predetermined defect occurs in the satellite signal, and performs visual positioning.

[0161] In this way, in this embodiment, visual positioning can be performed appropriately when the accuracy of satellite positioning decreases due to a situation where poor satellite signal reception is occurring. In particular, it is possible to easily determine whether a situation where a predetermined poor satellite signal reception is occurring occurs, based on the entry into an area where poor satellite signal reception is occurring or the number of observed positioning satellites.

[0162] Next, a description will be given of the schematic configuration of the pedestrian terminal 1 according to the third embodiment. Fig. 17 is a block diagram showing the schematic configuration of the pedestrian terminal 1. The schematic configuration of the roadside device 3 is the same as that of the first embodiment (see Fig. 5).

[0163] In this embodiment, the processor 18 of the pedestrian terminal 1 performs the poor satellite radio wave determination process P31. The rest is substantially the same as in the first embodiment (see FIG. 4).

[0164] In the satellite radio wave poorness determination process P31, the processor 18 determines whether or not a predetermined poor condition occurs in the satellite radio wave.

[0165] Specifically, processor 18 determines whether the pedestrian has entered an area with poor satellite reception based on map information and the pedestrian's position information. That is, it determines whether the pedestrian has entered an area with poor satellite reception based on the position information of the area with poor satellite reception registered in the map information and the position information of the pedestrian's current location acquired by satellite positioning (area entry determination process).

[0166] In addition, the processor 18 obtains information on observed positioning satellites (available positioning satellites) from the satellite positioning unit 11 and determines whether the required number of positioning satellites (the number that can ensure the required accuracy) can be observed (satellite number determination process).

[0167] In the positioning control process P6, the processor 18 instructs the execution of a visual positioning process when the result of the satellite radio wave poor judgment process P31 indicates that a specified poor satellite signal is present in the satellite radio waves, i.e., when a pedestrian enters an area with poor satellite signal or when the required number of positioning satellites cannot be observed.

[0168] Next, the operation procedure of the pedestrian terminal 1 according to the third embodiment will be described. Fig. 18 is a flow diagram showing the operation procedure of the pedestrian terminal 1. Note that the pedestrian terminal 1 performs the same processing as in the first embodiment (see Figs. 6(A), (B), and (C)). Also, the operation procedure of the roadside device 3 is the same as in the first embodiment (see Fig. 9).

[0169] In the pedestrian terminal 1, the processor 18 acquires the position information of the pedestrian from the satellite positioning unit 11 (ST131). Next, the processor 18 determines whether or not a predetermined defect appears in the satellite radio wave (ST151). At this time, it determines whether or not the pedestrian has entered an area with poor satellite radio wave reception (area entry determination process). It also determines whether or not the required number of positioning satellites (the number that can ensure the required accuracy) cannot be observed (satellite number determination process).

[0170] Here, if a situation occurs in which a specific problem occurs in the satellite radio waves, i.e., if a pedestrian enters an area with poor satellite radio wave reception or if the required number of positioning satellites cannot be observed (Yes in ST151), the processor 18 turns on the camera 12 (ST134) and performs visual positioning (ST135).

[0171] On the other hand, if there is no situation where a predetermined defect occurs in the satellite radio wave, that is, if the pedestrian has not entered an area with poor satellite radio wave reception or if the required number of positioning satellites can be observed (No in ST151), the processor 18 turns off the camera 12 (ST136). In this case, the positioning result of the satellite positioning is used as the position information of the pedestrian.

[0172] (Fourth embodiment) Next, a fourth embodiment will be described. Note that points not particularly mentioned here are the same as those of the previous embodiments. Fig. 19 is an explanatory diagram showing an overview of positioning control performed by a pedestrian terminal 1 according to the fourth embodiment. Fig. 20 is an explanatory diagram showing an overview of image relative positioning performed by a pedestrian terminal 1.

[0173] In the first embodiment, the pedestrian terminal 1 performs visual positioning when a predetermined change in the pedestrian state occurs based on the detection result of the state sensor 13. On the other hand, in the present embodiment, the pedestrian terminal 1 performs visual positioning when a predetermined change in the pedestrian state occurs based on the detection result of the state sensor 13, and if no predetermined change in the pedestrian state occurs and the pedestrian's moving speed is faster than a predetermined value, performs image relative positioning periodically at predetermined intervals (see FIG. 19).

[0174] In image relative positioning, the amount of movement of the pedestrian relative to the previous position is estimated (self-movement amount estimation) based on changes in the images captured by the camera 12, specifically, changes in feature points extracted from the captured images at each time, and the current position of the pedestrian is relatively estimated (see FIG. 20 ). Specifically, the feature points extracted from the previous captured image are compared with the feature points extracted from the current captured image to obtain the amount of change in corresponding feature points between the previous captured image and the current captured image, and the amount of movement relative to the position obtained in the previous positioning is calculated based on the amount of change in the feature points, thereby obtaining the current position. Note that this image relative positioning is a positioning technology adopted, for example, in Visual SLAM (Simultaneous Localization and Mapping) (registered trademark).

[0175] Here, visual positioning is absolute positioning, since it compares an image captured by camera 12 with a candidate image in the surrounding environment DB and acquires location information corresponding to a candidate image that has been successfully matched as location information for the pedestrian's current location. Since image matching is performed in this visual positioning, the processing load on processor 18 is heavy, resulting in high power consumption. On the other hand, image-relative positioning does not perform image matching like visual positioning, so the processing load on processor 18 is lighter and power consumption is lower than in visual positioning.

[0176] Furthermore, in image relative positioning, the camera 12 is turned on, which consumes more power than satellite positioning, but in this embodiment, image relative positioning is performed periodically at predetermined intervals. That is, image relative positioning is not performed continuously, but is performed intermittently. This makes it possible to reduce power consumption.

[0177] 19(B), when a pedestrian is running at a constant speed, there is no significant change in the pedestrian's state, but as the pedestrian's movement speed increases, image relative positioning is performed. On the other hand, when a pedestrian is running at a constant speed, the processing load of image relative positioning increases and there is no significant change in the direction of movement, so image relative positioning is performed intermittently, which reduces power consumption and ensures a certain degree of positional accuracy.

[0178] Next, a schematic configuration of the pedestrian terminal 1 according to the fourth embodiment will be described. Fig. 21 is a block diagram showing a schematic configuration of the pedestrian terminal 1.

[0179] In this embodiment, the processor 18 of the pedestrian terminal 1 performs the image relative positioning process P41. The rest is substantially the same as in the first embodiment (see FIG. 4).

[0180] In the image relative positioning process P41, the processor 18 estimates the relative movement amount of the pedestrian from the previous position (self-movement amount estimation) based on the change in the image captured by the camera 12, specifically, the change in the feature points extracted from the image captured at each time, and relatively estimates the current position of the pedestrian.

[0181] Meanwhile, in the pedestrian state determination process P5, similarly to the first embodiment, the processor 18 determines whether or not a predetermined change has occurred in the pedestrian state based on the detection result of the state sensor 13. In addition, in this embodiment, if no predetermined change has occurred in the pedestrian state, the processor 18 determines whether or not the moving speed of the pedestrian is faster than a predetermined value (speed determination process).

[0182] In the positioning control process P6, processor 18 instructs execution of visual positioning process P8 when a predetermined change occurs in the pedestrian's state. Furthermore, if the predetermined change does not occur in the pedestrian's state and the pedestrian's moving speed is greater than or equal to a predetermined value, processor 18 instructs execution of image relative positioning process P41. At this time, processor 18 determines whether or not a predetermined time has elapsed since the previous execution of image relative positioning process P41 (elapsed time determination process), and if the predetermined time has not yet elapsed, does not instruct execution of image relative positioning process P41. As a result, image relative positioning process P41 is executed intermittently at predetermined intervals.

[0183] Next, the operation procedure of the pedestrian terminal 1 according to the fourth embodiment will be described. Fig. 22 is a flow diagram showing the operation procedure of the pedestrian terminal 1. Note that the pedestrian terminal 1 performs the same processing as in the first embodiment (see Figs. 6(A), (B), and (C)). Also, the operation procedure of the roadside device 3 is the same as in the first embodiment (see Fig. 9).

[0184] In the pedestrian terminal 1, similarly to the first embodiment (see FIG. 7), the processor 18 acquires the position information of the pedestrian from the satellite positioning unit 11, and then the processor 18 performs the processes (ST131 to ST133) up to determining whether or not a predetermined change has occurred in the pedestrian state based on the pedestrian state information. Furthermore, if a predetermined change has occurred in the pedestrian state (Yes in ST133), similarly to the first embodiment, the processor 18 turns on the camera 12 (ST134) and performs visual positioning (ST135).

[0185] On the other hand, if the predetermined change in the pedestrian state does not occur (No in ST133), processor 18 next determines whether the movement speed of the pedestrian is equal to or greater than a predetermined value (ST161).

[0186] If the moving speed of the pedestrian is equal to or greater than the predetermined value (Yes in ST161), it is then determined whether a predetermined time has passed since the previous image relative positioning (ST162).

[0187] Here, if a predetermined time has passed since the previous image relative positioning (Yes in ST162), then the camera 12 is turned on (ST163) and image relative positioning is performed (ST164).

[0188] On the other hand, if the moving speed of the pedestrian is less than the predetermined value (No in ST161) or if the predetermined time has not elapsed since the previous image relative positioning (No in ST162), then the camera 12 is turned off (ST136). In this case, the positioning result of the satellite positioning is used as the position information of the pedestrian.

[0189] (Fifth embodiment) Next, a fifth embodiment will be described. Note that points not particularly mentioned here are the same as those in the above-described embodiments. Fig. 23 is an explanatory diagram showing an overview of positioning control performed by a pedestrian terminal 1 according to the fifth embodiment.

[0190] In the first embodiment, the pedestrian terminal 1 performs visual positioning when a predetermined change occurs in the pedestrian state based on the detection result of the state sensor 13. On the other hand, in the present embodiment, the pedestrian terminal 1 performs visual positioning when a predetermined change occurs in the pedestrian state and the degree of the change exceeds a predetermined range, performs image relative positioning when a predetermined change occurs in the pedestrian state and the degree of the change does not exceed a predetermined range, and performs non-image relative positioning when a predetermined change does not occur in the pedestrian state.

[0191] For example, if the change in the pedestrian's orientation (direction of travel) is equal to or greater than a predetermined threshold (e.g., 90 degrees) (see FIG. 23(A)), it is determined that the degree of change in the pedestrian's state has exceeded a predetermined range, and visual positioning is performed. On the other hand, if the change in the pedestrian's orientation is less than the threshold (see FIG. 23(B)), it is determined that the degree of change in the pedestrian's state has not exceeded a predetermined range, and image relative positioning is performed. Also, if there is only a slight change in the pedestrian's orientation (see FIG. 23(C)), it is determined that the predetermined change has not occurred in the pedestrian's state, and non-image relative positioning is performed.

[0192] In the image relative positioning, as in the fourth embodiment, the current position of the pedestrian is relatively estimated by estimating the relative movement amount of the pedestrian from the previous position based on the change in the image captured at each time by the camera 12. Note that this image relative positioning is a positioning technology adopted in, for example, Visual SLAM (registered trademark).

[0193] In non-image relative positioning, the current position of the pedestrian is relatively estimated by estimating the relative movement amount of the pedestrian from the previous position based on the detection results of the state sensor 13 (such as the gyro sensor 22 and the acceleration sensor 21). Note that this non-image relative positioning is a positioning technology adopted in, for example, PDR (Pedestrian Dead-Reckoning).

[0194] In this embodiment, even if a predetermined change occurs in the pedestrian's condition, if the degree of change is small, visual positioning is not performed, and instead image relative positioning is performed.This image relative positioning does not involve heavy processing such as image matching, so power consumption can be reduced.

[0195] In addition, in this embodiment, if there is no predetermined change in the pedestrian's condition, non-image relative positioning is performed, which may improve the accuracy of the location information.In addition, non-image relative positioning uses the condition sensor 13 but not the camera 12, and does not perform heavy processing such as image matching, so power consumption can be reduced.

[0196] In this embodiment, non-image relative positioning is performed when a predetermined change in the pedestrian's condition does not occur, but it is also possible that non-image relative positioning is not performed and location information obtained by satellite positioning is used as location information for the pedestrian's current location.

[0197] Next, a description will be given of the schematic configuration of the pedestrian terminal 1 according to the fifth embodiment. Fig. 24 is a block diagram showing the schematic configuration of the pedestrian terminal 1. The schematic configuration of the roadside device 3 is the same as that of the first embodiment (see Fig. 5).

[0198] In this embodiment, the processor 18 of the pedestrian terminal 1 performs the non-image relative positioning process P51. The rest is substantially the same as in the fourth embodiment (see FIG. 21).

[0199] In the non-image relative positioning process P51, the processor 18 estimates the relative movement amount of the pedestrian from the previous position (self-movement amount estimation) based on the detection results of the status sensor 13 (such as the gyro sensor 22 and the acceleration sensor 21), and relatively estimates the current position of the pedestrian.

[0200] In the image relative positioning process P41, the processor 18, as in the fourth embodiment, estimates the relative movement amount of the pedestrian from the previous position based on the change in the image captured by the camera 12 at each time (self-movement amount estimation), and relatively estimates the current position of the pedestrian.

[0201] Meanwhile, in the pedestrian state determination process P5, similarly to the first embodiment, the processor 18 determines whether or not a predetermined change has occurred in the pedestrian state based on the detection result of the state sensor 13. Furthermore, in this embodiment, when a predetermined change has occurred in the pedestrian state, the processor 18 determines whether or not the degree of change in the pedestrian state has exceeded a predetermined range.

[0202] In the positioning control process P6, if a predetermined change occurs in the pedestrian state and the degree of change in the pedestrian state exceeds a predetermined range, processor 18 instructs execution of a visual positioning process P8. If a predetermined change occurs in the pedestrian state and the degree of change in the pedestrian state does not exceed a predetermined range, processor 18 instructs execution of an image relative positioning process P41. If a predetermined change does not occur in the pedestrian state, processor 18 instructs execution of a non-image relative positioning process P51.

[0203] Next, the operation procedure of the pedestrian terminal 1 according to the fifth embodiment will be described. Fig. 25 is a flow diagram showing the operation procedure of the pedestrian terminal 1. Note that the pedestrian terminal 1 performs the same processing as in the first embodiment (see Figs. 6(A), (B), and (C)). Also, the operation procedure of the roadside device 3 is the same as in the first embodiment (see Fig. 9).

[0204] In the pedestrian terminal 1, similarly to the first embodiment (see FIG. 7), the processor 18 acquires the position information of the pedestrian from the satellite positioning unit 11, and then performs processing (ST131 to ST133) up to determining whether or not a predetermined change has occurred in the pedestrian state based on the pedestrian state information.

[0205] If a predetermined change in the pedestrian state occurs (Yes in ST133), processor 18 turns on camera 12 (ST134). Next, processor 18 determines whether the degree of change in the pedestrian state exceeds a predetermined range (ST171).

[0206] Here, if the degree of change in the pedestrian's state exceeds a predetermined range (Yes in ST171), processor 18 performs visual positioning (ST135).

[0207] On the other hand, if the degree of change in the state of the pedestrian does not exceed the predetermined range (No in ST171), the processor 18 performs image relative positioning (ST163).

[0208] Furthermore, if a predetermined change in the pedestrian state does not occur (No in ST133), the processor 18 turns off the camera 12 (ST136) and performs non-image relative positioning (ST151).

[0209] In determining whether a predetermined change occurs in the pedestrian state (ST133), processor 18 may make the determination using a predetermined threshold value related to the amount or rate of change in the pedestrian state. On the other hand, in determining whether the degree of change in the pedestrian state exceeds a predetermined range (ST171), processor 18 may make the determination using a threshold value larger than the threshold value used in determining whether a predetermined change occurs in the pedestrian state (ST133).

[0210] (Sixth embodiment) Next, a sixth embodiment will be described. Note that points not particularly mentioned here are the same as those in the above-mentioned embodiments. Fig. 26 is an explanatory diagram showing an overview of positioning control performed by a pedestrian terminal 1 according to the sixth embodiment.

[0211] In the first embodiment, the pedestrian terminal 1 performs visual positioning when a predetermined change occurs in the pedestrian state based on the detection result of the state sensor 13. On the other hand, in the present embodiment, the pedestrian terminal 1 performs visual positioning when a predetermined change occurs in the pedestrian state and the degree of the change exceeds a predetermined range, and performs route estimation positioning when a predetermined change occurs in the pedestrian state and the degree of the change does not exceed the predetermined range.

[0212] In route estimation positioning, the system determines which of the candidate route patterns corresponds to the route that the pedestrian is expected to have actually taken, based on the pedestrian's most recent trajectory, i.e., the pedestrian's position information at each time immediately preceding the pedestrian's time, and estimates the pedestrian's current location based on the corresponding route pattern.

[0213] Candidate route patterns are acquired using a machine learning model for route pattern recognition built by machine learning such as deep learning. During learning, information on the trajectories of each pedestrian's past travels, i.e., location information (positioning history information) that is the positioning results of each pedestrian at each time acquired by past positioning on each pedestrian terminal 1, is collected, and a machine learning model for pattern recognition is built using this as learning data. Note that the trajectory of each pedestrian is determined based on the configuration of roads that pedestrians can actually travel on, and can be classified into a corresponding number of route patterns.

[0214] In this case, in addition to the position information acquired by satellite positioning, position information acquired by non-image relative positioning (such as PDR) is used as training data. Position information acquired by visual positioning may also be used as training data. Position information acquired by image relative positioning (such as Visual SLAM (registered trademark)) may also be used as training data.

[0215] In addition, information that associates location information as a positioning result when positioning other than satellite positioning as standard positioning (non-image relative positioning, visual positioning, image relative positioning) is performed with location information as a positioning result of the satellite positioning at that time may be used as learning data.

[0216] Next, a schematic configuration of the pedestrian terminal 1 according to the sixth embodiment will be described. Fig. 27 is a block diagram showing a schematic configuration of the pedestrian terminal 1.

[0217] In this embodiment, the processor 18 of the pedestrian terminal 1 performs the route estimation positioning process P61. The rest is substantially the same as in the first embodiment (see FIG. 4).

[0218] In the route estimation and positioning process P61, the route pattern of the pedestrian is determined based on the pedestrian's most recent trajectory, that is, the pedestrian's position information at each time point immediately before the pedestrian, and the pedestrian's current position is estimated.

[0219] Meanwhile, in the pedestrian state determination process P5, similarly to the fifth embodiment, the processor 18 determines whether or not a predetermined change has occurred in the pedestrian state based on the detection result of the state sensor 13. Furthermore, if a predetermined change has occurred in the pedestrian state, the processor 18 determines whether or not the degree of change in the pedestrian state has exceeded a predetermined range.

[0220] In the positioning control process P6, if a predetermined change occurs in the pedestrian state and the degree of change in the pedestrian state exceeds a predetermined range, the processor 18 instructs the execution of a visual positioning process P8. In addition, if a predetermined change occurs in the pedestrian state and the degree of change in the pedestrian state does not exceed the predetermined range, the processor 18 instructs the execution of a route estimation positioning process P61.

[0221] Next, the operation procedure of the pedestrian terminal 1 according to the sixth embodiment will be described. Fig. 28 is a flow diagram showing the operation procedure of the pedestrian terminal 1. Note that the pedestrian terminal 1 performs the same processing as in the first embodiment (see Figs. 6(A), (B), and (C)). Also, the operation procedure of the roadside device 3 is the same as in the first embodiment (see Fig. 9).

[0222] In the pedestrian terminal 1, similarly to the first embodiment (see FIG. 7), the processor 18 acquires the position information of the pedestrian from the satellite positioning unit 11, and then performs processing (ST131 to ST133) up to determining whether or not a predetermined change has occurred in the pedestrian state based on the pedestrian state information.

[0223] Furthermore, if a predetermined change occurs in the pedestrian's condition (Yes in ST133), processor 18 determines whether the degree of change in the pedestrian's condition exceeds a predetermined range (ST171), and if the degree of change in the pedestrian's condition exceeds the predetermined range (Yes in ST171), processor 18 turns on camera 12 (ST134) and performs visual positioning (ST135).

[0224] On the other hand, if the degree of change in the pedestrian state does not exceed the predetermined range (No in ST171), the processor 18 turns off the camera 12 (ST163) and performs route estimation and positioning (ST181).

[0225] Furthermore, if there is no predetermined change in the state of the pedestrian (No in ST133), the processor 18 turns off the camera 12 (ST136).

[0226] (Seventh embodiment) Next, a seventh embodiment will be described. Note that points not particularly mentioned here are the same as those in the above-mentioned embodiments. Fig. 29 is an explanatory diagram showing an overview of positioning control performed by a pedestrian terminal 1 according to the seventh embodiment.

[0227] In the first embodiment, the pedestrian terminal 1 performs visual positioning when a predetermined change in the pedestrian state occurs based on the detection result of the state sensor 13. On the other hand, in the present embodiment, the pedestrian terminal 1 performs visual positioning when a predetermined change in the pedestrian state occurs based on the detection result of the state sensor 13, and also performs visual positioning when the pedestrian approaches a predetermined safe area even if no predetermined change in the pedestrian state occurs.

[0228] Here, a safe area is a place where the risk of a pedestrian colliding with a vehicle is low and the need for high positioning accuracy is relatively low. Specifically, safe areas include, for example, the inside of a vehicle (see Figure 29(A)), the inside of a building (see Figure 29(B)), and structures of pedestrian-only walkways such as footbridges and pedestrian decks (see Figure 29(C)).

[0229] In this embodiment, the pedestrian terminal 1 performs visual positioning and acquires highly accurate position information when the pedestrian approaches the safety area, i.e., immediately before entering the safety area, thereby enabling the pedestrian terminal 1 to confirm that the pedestrian is definitely entering the safety area.

[0230] In this case, for example, if the safety area is the inside of a vehicle, visual positioning is performed when the pedestrian approaches the vehicle. If the safety area is indoors, visual positioning is performed when the pedestrian approaches the entrance to a building. If the safety area is a building such as a pedestrian bridge, visual positioning is performed when the pedestrian approaches the entrance to the building.

[0231] In this embodiment, visual positioning continues from the time a pedestrian approaches a safety area until the pedestrian enters the safety area. Meanwhile, general photographed images of the interior of the safety area are registered as candidate images in the surrounding environment DB. In visual positioning, general photographed images of the interior of the safety area are extracted from the surrounding environment DB as candidates for comparison, and if image comparison is successful, it is confirmed that the pedestrian is staying within the safety area.

[0232] After performing visual positioning, the pedestrian terminal 1 can acquire the pedestrian's location information using satellite positioning if the pedestrian is outdoors, for example, while the pedestrian is staying in a safe area, and can acquire the pedestrian's location information using indoor positioning such as PDR if the pedestrian is indoors.

[0233] Next, a description will be given of the schematic configuration of the pedestrian terminal 1 according to the seventh embodiment. Fig. 30 is a block diagram showing the schematic configuration of the pedestrian terminal 1. The schematic configuration of the roadside device 3 is the same as that of the first embodiment (see Fig. 5).

[0234] In this embodiment, the processor 18 of the pedestrian terminal 1 performs the area proximity determination process P71. The rest is substantially the same as in the first embodiment (see FIG. 4).

[0235] In the area approach determination process P71, the processor 18 determines whether or not the pedestrian has approached a predetermined safety area based on map information. Specifically, based on the position information of the safety area registered in the map information and the position information of the pedestrian's current location acquired by satellite positioning, it is determined that the pedestrian is approaching the safety area if the pedestrian is located within a predetermined distance from the safety area and is moving in a direction approaching the safety area.

[0236] If the safe area is indoors, the location information of buildings included in the map information is used to determine whether the pedestrian has approached the safe area. In this case, visual positioning extracts general indoor images (candidate images) from the surrounding environment DB as targets for matching, and if image matching is successful, it is confirmed that the pedestrian is indoors.

[0237] Furthermore, if the safety area is inside a structure (fixed object) such as a footbridge, the location information of the structure included in the map information is used to determine whether the pedestrian has approached the safety area. In this case, in visual positioning, photographed images (candidate images) of the interior of a general structure are extracted from the surrounding environment DB as objects to be matched, and if the image match is successful, it is confirmed that the pedestrian is on the structure.

[0238] If the safety area is inside a vehicle, the vehicle's position information included in the vehicle information received from the in-vehicle terminal 2 via pedestrian-to-vehicle communication is used to determine whether the pedestrian has approached the safety area. In this case, in visual positioning, a typical image (candidate image) taken inside a vehicle is extracted from the surrounding environment DB as a matching target, and if the image matching is successful, it is confirmed that the pedestrian is inside the vehicle. In this case, the current position of the pedestrian may be set to the current position of the vehicle.

[0239] In the positioning control process P6, the processor 18 instructs the execution of a visual positioning process P8 when the determination result of the area approach determination process P71 is that the pedestrian has approached a predetermined safety area.

[0240] Next, the operation procedure of the pedestrian terminal 1 according to the seventh embodiment will be described. Fig. 31 is a flow diagram showing the operation procedure of the pedestrian terminal 1. Note that the pedestrian terminal 1 performs the same processing as in the first embodiment (see Figs. 6(A), (B), and (C)). Also, the operation procedure of the roadside device 3 is the same as in the first embodiment (see Fig. 9).

[0241] In the pedestrian terminal 1, similarly to the first embodiment (see FIG. 7), the processor 18 acquires the position information of the pedestrian from the satellite positioning unit 11, and then the processor 18 performs the processes (ST131 to ST133) up to determining whether or not a predetermined change has occurred in the pedestrian state based on the pedestrian state information. Furthermore, if a predetermined change has occurred in the pedestrian state (Yes in ST133), similarly to the first embodiment, the processor 18 turns on the camera 12 (ST134) and performs visual positioning (ST135).

[0242] On the other hand, if the predetermined change in the pedestrian state does not occur (No in ST133), then processor 18 determines whether the pedestrian has approached the safety area (ST191).

[0243] Here, if the pedestrian approaches the safety area (Yes in ST191), the processor 18 turns on the camera 12 (ST134) and performs visual positioning (ST135).

[0244] On the other hand, if the pedestrian is not approaching the safety area (No in ST191), the processor 18 turns off the camera 12 (ST136).

[0245] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0246] The pedestrian device, mobile device, positioning system, and positioning method according to the present invention have the effect of performing visual positioning to obtain highly accurate position information, while also being able to reduce power consumption and achieve power savings, and are useful as pedestrian devices that are held by pedestrians to perform positioning to measure the pedestrian's current position, mobile devices, positioning systems, and positioning methods that are held by moving bodies such as pedestrians and vehicles to perform positioning to measure the moving body's current position, etc. [Explanation of symbols]

[0247] 1 Pedestrian terminal (pedestrian device, mobile device, computer) 2. In-vehicle terminal 3 Roadside unit 11 Satellite positioning unit 12 Camera 13 Status Sensor 17. Memory 18 processors 21 Acceleration sensor 22 Gyro sensor 23 Geomagnetic sensor 24 Eye-gaze sensor 25 Front camera 26 Foot Camera

Claims

1. an external sensor that detects objects around the pedestrian; a memory that stores surrounding environment information relating to the surrounding environment of a pedestrian; a state sensor for detecting a state of a pedestrian; a processor, The processor: Standard positioning is performed to obtain the location information of pedestrians, When a predetermined change in the state of a pedestrian occurs based on the detection result of the state sensor, the occurrence of the specific event is determined as a specific event, and when the specific event occurs, the system performs visual positioning to estimate the current position of the pedestrian by comparing the detected image of the external sensor with the surrounding environment information, and The pedestrian device is characterized in that the external sensor is controlled to be turned on when the visual positioning is being performed, and to be turned off otherwise.

2. The pedestrian device according to claim 1 , wherein the external sensor is one of a camera and a LIDAR.

3. the state sensor is an acceleration sensor, The processor:

2. The pedestrian device according to claim 1, wherein the visual positioning is performed when a predetermined change in the moving speed of the pedestrian occurs as the specific event based on the detection result of the acceleration sensor.

4. the state sensor is a direction sensor, The processor:

2. The pedestrian device according to claim 1, wherein the visual positioning is performed when a predetermined change in the pedestrian's orientation occurs as the specific event based on the detection result of the orientation sensor.

5. Equipped with a gaze sensor that detects the gaze of pedestrians, The processor:

2. The pedestrian device according to claim 1, wherein the visual positioning is performed when a predetermined change in the pedestrian's line of sight appears as the specific event based on the detection result of the line of sight sensor.

6. The processor:

2. The pedestrian device according to claim 1, wherein the specific event requiring the visual positioning is detected based on the detection results of the plurality of state sensors.

7. An external sensor that detects objects around a pedestrian; a memory that stores surrounding environment information relating to the surrounding environment of a pedestrian; a processor, The processor: Satellite positioning is performed as standard positioning to obtain pedestrian location information, acquiring a trajectory of a pedestrian based on the position information acquired by the satellite positioning; When a predetermined change appears in the trajectory, the occurrence of the specific event is determined as a specific event, and when the specific event occurs, visual positioning is performed to estimate the current position of the pedestrian by comparing the detected image of the external sensor with the surrounding environment information, and The pedestrian device is characterized in that the external sensor is controlled to be turned on when the visual positioning is being performed, and to be turned off otherwise.

8. An external sensor that detects objects around a pedestrian; a memory that stores surrounding environment information relating to the surrounding environment of a pedestrian; a processor, The processor: Satellite positioning is performed as standard positioning to obtain pedestrian location information, A specific event is defined as a situation in which a predetermined defect occurs in the satellite radio wave. When the specific event occurs, visual positioning is performed to estimate the current position of the pedestrian by comparing the detected image of the external sensor with the surrounding environment information. The pedestrian device is characterized in that the external sensor is controlled to be turned on when the visual positioning is being performed, and to be turned off otherwise.

9. The processor:

9. The pedestrian device according to claim 8, wherein the visual positioning is performed when the device enters an area with poor satellite reception registered in the map information, based on map information and position information of the pedestrian.

10. An external sensor that detects objects around a pedestrian; A foot camera captures the road surface under pedestrians' feet, a memory that stores surrounding environment information relating to the surrounding environment of a pedestrian; a processor, The processor: Standard positioning is performed to obtain the location information of pedestrians, When a specific event occurs, visual positioning is performed to estimate the current position of the pedestrian by comparing the detected image of the external sensor with the surrounding environment information, and The external sensor is turned on when the visual positioning is performed, and is turned off otherwise; The visual positioning of the pedestrian device is characterized in that it executes a first matching process of matching the detected image of the external sensor with the surrounding environment information, and a second matching process of matching the image captured by the foot camera with the surrounding environment information, and estimates the current position of the pedestrian based on the matching results of the first matching process and the second matching process.

11. The processor:

2. The pedestrian device according to claim 1, wherein even if a predetermined change in the pedestrian state does not appear, if the pedestrian's movement speed is faster than a predetermined value, image relative positioning is performed at predetermined intervals to estimate the pedestrian's relative movement amount from the previous position based on a change in the image detected by the external sensor at each time, and to relatively estimate the pedestrian's current position.

12. The processor: performing the visual positioning when a predetermined change occurs in the pedestrian state and the degree of the change exceeds a predetermined range; 2. The pedestrian device according to claim 1, wherein, when a predetermined change occurs in the pedestrian state and the degree of change does not exceed a predetermined range, the pedestrian device performs image relative positioning to estimate a relative movement amount of the pedestrian from a previous position based on a change in the image detected by the external sensor at each time, and to relatively estimate a current position of the pedestrian.

13. The processor:

13. The pedestrian device according to claim 12, wherein, when a predetermined change in the pedestrian state does not occur, non-image relative positioning is performed to estimate a relative movement amount of the pedestrian from a previous position based on a detection result of the state sensor, and to relatively estimate a current position of the pedestrian.

14. The processor: performing the visual positioning when a predetermined change occurs in the pedestrian state and the degree of the change exceeds a predetermined range; 2. The pedestrian device according to claim 1, wherein, when a predetermined change in the pedestrian state occurs and the degree of the change does not exceed a predetermined range, route estimation positioning is performed to determine the pedestrian's route pattern based on the pedestrian's previous trajectory and estimate the pedestrian's current position.

15. The processor:

2. The pedestrian device according to claim 1, wherein the visual positioning is performed when the pedestrian approaches a predetermined safety area even if a predetermined change in the pedestrian state does not occur.

16. A positioning system configured with one or more computers that executes a process for acquiring position information of a mobile body in a mobile body device, The vehicle device includes an external sensor that detects objects around the vehicle, and a state sensor that detects the state of a pedestrian, The computer a memory for storing surrounding environment information relating to the surrounding environment of the moving body; Standard positioning is performed to obtain the location information of the moving object, When a predetermined change in the state of a pedestrian occurs based on the detection result of the state sensor, the state sensor determines that this is a specific event, and when the specific event occurs, the system performs visual positioning to estimate the current position of the moving object by comparing the detected image of the external sensor with the surrounding environment information, and A positioning system characterized in that the external sensor is controlled to be turned on when the visual positioning is performed and to be turned off otherwise.

17. A positioning method in which one or more computers execute a process for acquiring position information of a mobile body in a mobile body device, The computer Standard positioning is performed to obtain the location information of the moving object, Based on the detection results of a status sensor that detects the status of a pedestrian, a specific event is determined to be a predetermined change in the status of the pedestrian. When the specific event occurs, visual positioning is performed to estimate the current position of the mobile body by comparing an image detected by an external sensor that detects objects around the mobile body with surrounding environment information related to the surrounding environment of the mobile body, and A positioning method characterized in that the external sensor is controlled to be turned on when the visual positioning is performed and turned off otherwise.

Images