Route prediction device, in-vehicle device equipped with the same, route prediction system, route prediction method, and computer program

Abstract

To provide a route prediction device that can predict a pedestrian's path of movement earlier before the pedestrian actually takes action, not only at a specific location such as a pedestrian crossing, etc., an in-vehicle device therewith, a route prediction system, a route prediction method, and a computer program.SOLUTION: A route prediction device includes: a collection unit that collects sensor data acquired by at least one of in-vehicle sensors and roadside sensors via wireless or wired communication; an analysis unit that detects a moving object from the sensor data collected by the collection unit and identifies the positions and signs of the moving object; a prediction unit that predicts a route including a future location of the moving object using the positions and signs of the moving object identified by the analysis unit; a storage unit that stores the positions and indications identified by the analyzing section, corresponding to each moving object; and a learning unit that trains the prediction unit using the positions and signs stored in the storage unit for a predetermined period of time.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to a route prediction device, an in-vehicle device including the same, a route prediction system, a route prediction method, and a computer program.

Background Art

[0002] There has been proposed a system that uploads sensor information from a fixedly installed sensor such as a street surveillance camera (hereinafter also referred to as an infrastructure sensor) to a server computer (hereinafter simply referred to as a server), analyzes it, and monitors it. In addition, various sensors are also mounted on automobiles, motorcycles, etc. (hereinafter referred to as vehicles), and it has been proposed to upload this sensor information to a server for analysis and use it for driving support. It has also been proposed to combine the analysis results of in-vehicle sensor data and the analysis results of infrastructure sensor data for use in driving support.

[0003] Patent Document 1 cited below discloses a system that appropriately distributes driving support information to a vehicle and prevents a collision accident when there is a risk of collision between a pedestrian or bicycle attempting to cross a road and the vehicle. In this system, a pedestrian or bicycle is sensed by a sensor, their behavior is predicted, and the possibility of collision with the vehicle is determined. In addition, a sensor installed on the roadside is used to predict the route of a pedestrian or vehicle in combination with traffic signal information and road surface information. In predicting the route of a pedestrian, prediction is performed using the positional relationship with a crosswalk and a motion model of uniform linear motion.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Patent Document 1, when predicting a pedestrian's route, a plurality of possible routes are presented as candidates. However, in driving assistance, it is preferable that the predicted routes be fewer. Further, since the prediction is made only from a motion model based on the pedestrian's position, geographical conditions, speed, etc., there is a problem that it is not known which route the actual pedestrian will select until the pedestrian starts to cross a crosswalk or just before starting to cross. It is desirable to be able to limit the pedestrian's route earlier.

[0006] In Patent Document 1, it is premised that the pedestrian crosses only on the crosswalk, and there is a problem that it is impossible to predict other dangerous crossings that are difficult for the driver to handle. Further, since the moving object is detected only by a road condition grasping sensor provided on the roadside, there is a problem that prediction is impossible outside the detection range of the roadside sensor.

[0007] Therefore, an object of the present disclosure is to provide a route prediction device, an in-vehicle device including the same, a route prediction system, a route prediction method, and a computer program that can predict a pedestrian's movement route earlier before the pedestrian actually acts, not limited to a specific place such as a crosswalk.

Means for Solving the Problems

[0008] A route prediction device according to an aspect of the present disclosure includes a collection unit that collects sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication, an analysis unit that detects a moving object from the sensor data collected by the collection unit and specifies the position and signs of the moving object, a prediction unit that predicts a route including the future position of the moving object using the position and signs of the moving object specified by the analysis unit, a storage unit that stores the position and signs specified by the analysis unit in correspondence with each moving object, and a learning unit that learns the prediction unit using the position and signs stored in the storage unit for a predetermined period.

[0009] A route prediction system according to another aspect of the present disclosure includes an in-vehicle device mounted on a vehicle including a first sensor, a roadside device including a second sensor installed on the road, and a route prediction device. The in-vehicle device transmits sensor data acquired by the first sensor to the prediction device via wireless communication. The roadside device transmits sensor data acquired by the second sensor to the prediction device via wireless communication or wired communication. The route prediction device includes a receiving unit that receives the sensor data transmitted from the in-vehicle device and the sensor data transmitted from the roadside device, an analysis unit that detects a moving object from the sensor data received by the receiving unit and identifies the position and signs of the moving object, a prediction unit that predicts a route including the future position of the moving object using the position and signs of the moving object identified by the analysis unit, a storage unit that stores the position and signs identified by the analysis unit in association with each moving object, and a learning unit that learns the route prediction unit using the position and signs stored in the storage unit for a predetermined period.

[0010] A route prediction system according to still another aspect of the present disclosure includes an in-vehicle device mounted on a vehicle including a sensor, and a distribution device that distributes predetermined information to the in-vehicle device via wireless communication. The in-vehicle device includes a receiving unit that receives the predetermined information, an analysis unit that detects a moving object from the sensor data detected by the sensor and identifies the position and signs of the moving object, and a prediction unit that predicts a route including the future position of the moving object using the position and signs of the moving object identified by the analysis unit and the predetermined information received by the receiving unit. The predetermined information is learning parameters obtained by learning a predictor of the same model as the prediction unit using a set of positions and signs obtained by repeatedly performing, for a predetermined period, a process of detecting a moving object from sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor and identifying the position and signs of the moving object.

[0011] An in-vehicle device according to still another aspect of the present disclosure includes a sensor, an analysis unit that detects a moving object from sensor data detected by the sensor and identifies the position and signs of the moving object, and the position and signs of the moving object identified by the analysis unit, and a prediction unit that predicts a route including the future position of the moving object using predetermined information received by the receiving unit, and a storage unit that stores learning parameters used by the prediction unit for route prediction for each predetermined area including roads on which the vehicle can travel, where the learning parameters are obtained by learning a predictor of the same model as the prediction unit using a set of positions and signs obtained by repeatedly performing, for a predetermined period, a process of detecting a moving object from sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor and identifying the position and signs of the moving object.

[0012] A route prediction method according to still another aspect of the present disclosure includes a collection step of collecting sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication, an analysis step of detecting a moving object from the sensor data collected in the collection step and identifying the position and signs of the moving object, a prediction step of predicting a route including the future position of the moving object using the position and signs of the moving object identified in the analysis step, a storage step of storing the position and signs identified in the analysis step in correspondence with each moving object, and a learning step of learning the prediction step using the position and signs stored for a predetermined period in the storage step.

[0013] A computer program according to still another aspect of the present disclosure causes a computer to implement a collection function that collects sensor data acquired by at least one of in-vehicle sensors and roadside sensors via wireless communication or wired communication, an analysis function that detects a moving object from the sensor data collected by the collection function and identifies the position and signs of the moving object, a prediction function that predicts a route including the future position of the moving object using the position and signs of the moving object identified by the analysis function, a storage function that stores the position and signs identified by the analysis function in association with each moving object, and a learning function that learns the prediction function using the position and signs stored by the storage function for a predetermined period.

[0014] Note that part or all of each of the route prediction device, the in-vehicle device, and the route prediction system can be realized as a semiconductor integrated circuit. Further, it can also be realized as a computer-readable recording medium (such as a USB memory, a memory card (such as an SD card), a magnetic disk (such as an HDD), a magneto-optical disk, an optical disk (such as a DVD disk, a Blue-ray disk, etc.)) recording the above computer program.

Advantages of the Invention

[0015] According to the present disclosure, it is possible to predict the movement route of a pedestrian earlier before the pedestrian actually acts, not limited to a specific place such as a crosswalk.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

[0017] [Description of Embodiments of the Present Disclosure] First, the contents of the embodiments of the present disclosure will be listed and described. At least a part of the embodiments described below may be arbitrarily combined.

[0018] (1) The information providing apparatus according to the first aspect of the present disclosure includes: a collection unit that collects sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication; an analysis unit that detects a moving object from the sensor data collected by the collection unit and identifies the position and signs of the moving object; a prediction unit that predicts a route including the future position of the moving object using the position and signs of the moving object identified by the analysis unit; a storage unit that stores the position and signs identified by the analysis unit in association with each moving object; and a learning unit that learns the prediction unit using the position and signs stored in the storage unit for a predetermined period. Thereby, the moving route of a moving object such as a pedestrian can be predicted earlier before the moving object actually acts.

[0019] (2) Preferably, the moving object is a person, and the signs include information representing the direction of the person's face. Thereby, the accuracy of route prediction for a person (such as a pedestrian) can be improved.

[0020] (3) More preferably, the signs further include information representing the direction of the person's body. Thereby, the accuracy of route prediction for a person (such as a pedestrian) can be further improved.

[0021] (4) Even more preferably, the learning unit learns the prediction unit by machine learning, and among the time series data of the position and signs identified for each moving object over a predetermined time and stored in the storage unit, the first time series data of the position and signs belonging to the more past is used as the input data for machine learning, and the time series data that does not include the first time series data among the time series data of the position and signs identified for each moving object over a predetermined time and stored in the storage unit is used as the teacher data for machine learning. Thereby, data for machine learning can be automatically generated, and the prediction unit can be efficiently learned.

[0022] (5) Preferably, the route prediction apparatus further includes a distribution unit that distributes the route predicted by the prediction unit to an in-vehicle device. Thereby, the in-vehicle device can use the moving route as driving support information.

[0023] More preferably, the distribution unit distributes the image data of the moving object corresponding to the route together with the route predicted by the prediction unit. As a result, the in-vehicle device can present the image of the moving object such as a pedestrian, which is the target of the predicted route, to the driver so that the driver can visually confirm it, and thus the reliability of the received route can be confirmed.

[0024] Even more preferably, the route prediction device further includes an evaluation unit that evaluates the possibility of collision between the moving object to be predicted on the route and another moving object different from the moving object using the route predicted by the prediction unit. As a result, the possibility of collision can be presented and the collision can be prevented.

[0025] Preferably, the collection unit collects the sensor data acquired by the in-vehicle sensor and the sensor data acquired by the roadside sensor, and further includes a determination unit that determines whether the first moving object detected from the sensor data acquired by the in-vehicle sensor and the second moving object detected from the sensor data acquired by the roadside sensor are the same moving object. The prediction unit predicts the route using both the positions and signs specified for each of the first moving object and the second moving object determined to be the same moving object. As a result, the moving route of the moving object can be predicted even earlier before the moving object such as a pedestrian actually acts.

[0026] The in-vehicle device according to the second aspect of the present disclosure is an in-vehicle device including the above-described route prediction device. As a result, the in-vehicle device can predict the moving route of the moving object earlier before the moving object such as a pedestrian actually acts.

[0027] (10) The route prediction system according to the third aspect of the present disclosure includes an in-vehicle device mounted on a vehicle including a first sensor, a roadside device including a second sensor installed on the road, and a route prediction device. The in-vehicle device transmits sensor data acquired by the first sensor to the route prediction device via wireless communication. The roadside device transmits sensor data acquired by the second sensor to the route prediction device via wireless communication or wired communication. The route prediction device includes a receiving unit that receives the sensor data transmitted from the in-vehicle device and the sensor data transmitted from the roadside device, an analysis unit that detects a moving object from the sensor data received by the receiving unit and identifies the position and signs of the moving object, a prediction unit that predicts a route including the future position of the moving object using the position and signs of the moving object identified by the analysis unit, a storage unit that stores the position and signs identified by the analysis unit in association with each moving object, and a learning unit that learns the prediction unit using the position and signs stored in the storage unit for a predetermined period. Thereby, the in-vehicle device alone can predict the moving route of a moving object such as a pedestrian earlier before the moving object actually acts.

[0028] (11) The route prediction system according to the fourth aspect of the present disclosure includes an in-vehicle device mounted on a vehicle including a sensor, and a distribution device that distributes predetermined information to the in-vehicle device via wireless communication. The in-vehicle device includes a receiving unit that receives the predetermined information, an analysis unit that detects a moving object from the sensor data detected by the sensor and identifies the position and signs of the moving object, and a prediction unit that predicts a route including the future position of the moving object using the position and signs of the moving object identified by the analysis unit and the predetermined information received by the receiving unit. The predetermined information is learning parameters obtained by learning a predictor of the same model as the prediction unit using a set of positions and signs obtained by repeatedly performing, for a predetermined period, a process of detecting a moving object from sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor and identifying the position and signs of the moving object. Thereby, before a moving object such as a pedestrian actually acts, the moving route of the moving object can be predicted accurately earlier using appropriate predetermined information (parameters).

[0029] (12) Preferably, the route prediction system includes a plurality of distribution devices, each of the plurality of distribution devices being arranged for each predetermined area including roads on which vehicles can travel, and distributing predetermined information corresponding to the predetermined area via wireless communication. The receiving unit of the in-vehicle device receives the predetermined information corresponding to the predetermined area from the distribution device arranged in the predetermined area including the current position of the vehicle among the plurality of distribution devices. Thereby, it is possible to suppress the in-vehicle device from storing unnecessary data.

[0030] (13) More preferably, the in-vehicle device further includes a transmitting unit that transmits a transmission request for requesting the transmission of the predetermined information corresponding to the predetermined area to the distribution device arranged in the predetermined area including the current position of the vehicle. The distribution device that has received the transmission request transmits the predetermined information corresponding to the predetermined area in which the distribution device is arranged to the in-vehicle device that has transmitted the request. Thereby, it is possible to suppress the transmission of unnecessary predetermined information from the distribution device to the in-vehicle device.

[0031] (14) More preferably, the receiving unit receives the version of the predetermined information corresponding to the predetermined area distributed from the distribution device arranged in the predetermined area including the current position of the vehicle. The in-vehicle device includes a storage unit that stores learning parameters used by the prediction unit for route prediction for each predetermined area, and a determination unit that determines whether the version acquired by the receiving unit is newer than the version of the learning parameters corresponding to the predetermined area including the current position of the vehicle among the learning parameters stored in the storage unit. In response to the determination unit determining that the version acquired by the receiving unit is newer than the version of the learning parameters corresponding to the predetermined area including the current position of the vehicle among the learning parameters stored in the storage unit, the transmitting unit transmits a transmission request. Thereby, it is possible to further suppress the transmission of unnecessary predetermined information from the distribution device to the in-vehicle device.

[0032] (15) The in-vehicle device according to the fifth aspect of the present disclosure includes a sensor, an analysis unit that detects a moving object from sensor data detected by the sensor and specifies the position and signs of the moving object, and a prediction unit that predicts a path including the future position of the moving object from the position and signs of the moving object specified by the analysis unit. The in-vehicle device further includes a storage unit that stores learning parameters used by the prediction unit for predicting the path for each predetermined area including roads where the vehicle can pass. The learning parameters are obtained by repeatedly performing, for a predetermined period, a process of detecting a moving object from sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor and specifying the position and signs of the moving object, and using a set of positions and signs thus obtained to train a predictor having the same model as the prediction unit. Thereby, the in-vehicle device alone can accurately predict the moving path of a moving object, such as a pedestrian, earlier before the moving object actually moves, using appropriate predetermined information (parameters) according to the region.

[0033] (16) The path prediction method according to the sixth aspect of the present disclosure includes a collection step of collecting sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication, an analysis step of detecting a moving object from the sensor data collected in the collection step and specifying the position and signs of the moving object, a prediction step of predicting a path including the future position of the moving object using the position and signs of the moving object specified in the analysis step, a storage step of storing the position and signs specified in the analysis step in association with each moving object, and a learning step of training the prediction step using the position and signs stored for a predetermined period in the storage step. Thereby, the moving path of a moving object, such as a pedestrian, can be predicted earlier before the moving object actually moves.

[0034] (17) A computer program according to a seventh aspect of the present disclosure causes a computer to implement a collection function of collecting sensor data acquired by at least one of in-vehicle sensors and roadside sensors via wireless communication or wired communication, an analysis function of detecting a moving object from the sensor data collected by the collection function and specifying the position and signs of the moving object, a prediction function of predicting a route including the future position of the moving object using the position and signs of the moving object specified by the analysis function, a storage function of associating and storing the position and signs specified by the analysis function for each moving object, and a learning function of learning the prediction function using the position and signs stored for a predetermined period by the storage function. Thereby, it is possible to predict the moving route of a moving object such as a pedestrian earlier before the moving object actually acts.

[0035] [Details of Embodiments of the Present Disclosure] In the following embodiments, the same parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0036] (First Embodiment) [Overall Configuration] Referring to FIG. 1, a route prediction system 100 according to a first embodiment of the present disclosure includes a route prediction device 102, a roadside sensor 104 fixedly installed on a road and its surroundings (hereinafter also referred to as on-road), a vehicle 106, a mobile communication base station 108, a network 110, and a traffic signal (pedestrian signal) 112 for road traffic. A pedestrian 200 is a detection target of sensors mounted on the roadside sensor 104 and the vehicle 106. It is assumed that communication between the elements constituting the route prediction system 100 is performed via the base station 108 or by direct wireless communication or wired communication without passing through the base station 108. The base station 108 provides mobile communication services via an LTE (Long Term Evolution) line and a 5G (fifth generation mobile communication system) line.

[0037] The in-vehicle device mounted on the vehicle 106 has a communication function using an LTE line, a 5G line, or the like. The roadside sensor 104 is a device equipped with an imaging function installed on the road and its surroundings, and has a communication function with the base station 108. The roadside sensor 104 is, for example, a digital surveillance camera.

[0038] Referring to FIG. 2, the route prediction system 100 is arranged in an area including, for example, an intersection. The route prediction device 102 receives and analyzes sensor information (such as moving image data; hereinafter also referred to as sensor data) transmitted from the roadside sensor 104 and the in-vehicle device of the vehicle 106, detects the pedestrians 200 and 202, and predicts the movement routes 210 and 212 for each detected pedestrian. The predicted movement route (hereinafter referred to as the predicted route) is distributed (for example, broadcast) to the in-vehicle device. Vehicles 114, 116, and 118 traveling on the surrounding roads can use the received predicted route as driving support information.

[0039] FIGS. 1 and 2 typically show one base station 108 and one infrastructure sensor (roadside sensor 104), but usually, a plurality of base stations and a plurality of infrastructure sensors are provided. The route prediction device 102 communicates with more infrastructure sensors and in-vehicle devices of vehicles, collects and analyzes sensor information, and uses it for predicting movement routes.

[0040] [Hardware Configuration of Route Prediction Device] Referring to FIG. 3, the route prediction device 102 includes a control unit 120 that controls each part, a memory 122 that stores data, a communication unit 124 that performs communication, and a bus 126 for exchanging data between each part. The control unit 120 includes a CPU (Central Processing Unit) and realizes functions described later by controlling each part. The memory 122 includes a rewritable semiconductor non-volatile memory and a large-capacity storage device such as a hard disk drive (hereinafter referred to as HDD). The communication unit 124 receives sensor information (image data) uploaded from the roadside sensor 104 arranged on the road and sensor information uploaded from in-vehicle devices such as the vehicle 106. The data received by the communication unit 124 is transmitted to and stored in the memory 122. Thereby, the route prediction device 102 realizes a route prediction function as described later.

[0041] [Hardware Configuration and Function of In-Vehicle Device] Referring to FIG. 4, an example of the hardware configuration of the in-vehicle device 140 mounted on the vehicle 106 is shown. The in-vehicle device 140 includes an interface unit (hereinafter referred to as I / F unit) 144 connected to a sensor unit 142 mounted on the vehicle 106, a communication unit 146 that performs wireless communication, a memory 148 that stores data, a control unit 150 that controls them, a presentation unit 152 that presents information, and a bus 154 for exchanging data between each part.

[0042] The sensor unit 142 is a known video imaging device (e.g., a digital camera (CCD camera, CMOS camera)) mounted on the vehicle 106. The sensor unit 142 outputs a predetermined video signal (analog signal or digital data). The signal from the sensor unit 142 is input to the I / F unit 144. The I / F unit 144 includes an A / D conversion unit, samples an analog signal at a predetermined frequency when the analog signal is input, and generates and outputs digital data (sensor information). The generated digital data is transmitted to and stored in the memory 148. If the output signal from the sensor unit 142 is digital data, the I / F unit 144 stores the input digital data in the memory 148. The memory 148 is, for example, a rewritable non-volatile semiconductor memory or an HDD.

[0043] The communication unit 146 has a mobile communication function such as an LTE line and a 5G line, and communicates with the route prediction device 102. The communication between the in-vehicle device 140 and the route prediction device 102 is performed via the base station 108 or by direct wireless communication without passing through the base station 108. The communication unit 146 is composed of an IC for performing modulation and multiplexing adopted in each of the LTE line and the 5G line, an antenna for radiating and receiving radio waves of a predetermined frequency, and an RF circuit, etc. Since the communication unit 146 has a plurality of mobile communication functions such as an LTE line and a 5G line, when the vehicle 106 is traveling in an area where the 5G line communication service is not provided, communication by the LTE line becomes possible.

[0044] The presentation unit 152 includes an image display device such as a liquid crystal display and an acoustic device such as a speaker, and displays driving support information as an image and outputs it as sound. The control unit 150 includes a CPU and realizes the functions of the in-vehicle device 140 by controlling each unit. For example, the control unit 150 performs processes such as receiving a predicted route of a moving object (such as a pedestrian) from the route prediction device 102 and controlling the driving of the vehicle 106 and providing information for assisting the driver. Also, the control unit 150 acquires the current position of the vehicle 106 by GPS.

[0045] [Hardware Configuration and Functions of Infrared Sensors] The roadside sensor 104, which is an infrared sensor, is basically configured in the same way as the in-vehicle device 140. Referring to FIG. 5, an example of the hardware configuration of the roadside sensor 104 is shown. The roadside sensor 104 includes an I / F unit 162 connected to the sensor unit 160, a communication unit 164 that performs wireless communication or wired communication, a memory 166 that stores data, a control unit 168 that controls them, and a bus 170 for exchanging data between the units.

[0046] The sensor unit 160 is a known imaging device for video images (e.g., a digital camera). The sensor unit 160 outputs a predetermined video signal. The signal from the sensor unit 160 is input to the I / F unit 162. The I / F unit 162 includes an A / D conversion unit, and when an analog signal is input, it generates and outputs digital data (sensor information). The generated digital data is transmitted to and stored in the memory 166. If the output signal from the sensor unit 160 is digital data, the I / F unit 162 stores the input digital data in the memory 166. The memory 166 is, for example, a rewritable non-volatile semiconductor memory or an HDD.

[0047] The communication unit 164 has a mobile communication function and communicates with the route prediction device 102 by direct wireless communication or wired communication via the base station 108 or without passing through the base station 108. Since the roadside sensor 104 is fixedly installed, it does not need to support a plurality of mobile communication methods, and it is sufficient to support the mobile communication method provided by a nearby base station 108 (e.g., an LTE line or a 5G line). The communication unit 164 is composed of an IC for performing the adopted modulation and multiplexing, an antenna for radiating and receiving radio waves of a predetermined frequency, and an RF circuit, etc. Note that the communication function of the fixedly installed roadside sensor 104 is not limited to the case of passing through the base station 108 and is arbitrary. It may be a communication function using a wired LAN or a wireless LAN such as WiFi. In the case of WiFi communication, a device (such as a wireless router) that provides a WiFi service is provided separately from the mobile communication base station 108, and the roadside sensor 104 communicates with the route prediction device 102 via the network 110.

[0048] The control unit 168 includes a CPU and realizes the functions of the in-vehicle device 140 by controlling each unit. That is, the control unit 168 reads out the moving image data captured by the sensor unit 160 and stored in the memory 166 at a predetermined time interval, generates packet data, and transmits it from the communication unit 164 to the route prediction device 102.

[0049] [Functional Configuration of Route Prediction Device] With reference to FIG. 6, the functions of the route prediction device 102 will be described. The route prediction device 102 includes a packet receiving unit 180 that receives packet data, a packet transmitting unit 182 that transmits packet data, an analysis processing unit 184 that executes a predetermined analysis process on the received sensor information, a learning unit 186 that performs learning of route prediction, and a prediction unit 188 that executes a process of predicting the future movement route (predicted route) of a moving object. The packet receiving unit 180 receives packet data including sensor data from the roadside sensor 104 and the in-vehicle device 140. The received sensor data is input to the analysis processing unit 184.

[0050] The analysis processing unit 184 executes an analysis process on the input sensor data (such as moving image data) to detect a moving object. Here, it is assumed that the detection target is a moving pedestrian, and the "pedestrian" is not limited to a person walking, but means a person moving at an arbitrary speed and also includes a running person. If the uploaded moving image data includes a plurality of pedestrians, each pedestrian is detected. For the analysis process, known object detection processes including a process of obtaining the difference between frames and a feature amount extraction process can be used. For example, feature amounts (detection time, position, moving speed, moving direction, size, shape, color, etc.) regarding the detected moving object can be obtained. Thereby, a pedestrian can be distinguished from other moving objects, and it can also be determined whether they are the same pedestrian or different pedestrians.

[0051] Furthermore, for each detected pedestrian, the analysis processing unit 184 detects the position and signs as feature amounts. The "signs" mean information for predicting the future position of the detected pedestrian. Here, as signs, the orientation of the body and the orientation of the face of the detected pedestrian are detected. The method for detecting the orientation of the body and the orientation of the face is arbitrary. For example, a detection method using HOG (Histograms of Oriented Gradients) feature amounts or Haar-Like feature amounts can be used. Also, a detection method combining one of HOG feature amounts and Haar-Like feature amounts with AdaBoost (Adaptive Boosting) may be used. Also, a detection method using a convolutional neural network may be used. As an example, FIG. 7 shows a part of the image data of one frame. In FIG. 7, target frames 220 and 230 representing the detected pedestrians are displayed. For example, regarding the target frame 220, a face region (indicated by a face region frame 222) is specified, and the orientation of the face 224 is calculated by analyzing the face region. Also, regarding the target frame 220, a body region (indicated by a body region frame 226) is specified, and the orientation of the body 228 is calculated by analyzing the body region. Each of the face orientation 224 and the body orientation 228 is calculated as, for example, a two-dimensional vector in the horizontal plane. The analysis processing unit 184 outputs the analysis results (the position, the orientation of the body, and the orientation of the face for each pedestrian) to the learning unit 186 and the prediction unit 188.

[0052] When the analysis results (the position, the orientation of the body, and the orientation of the face for each pedestrian) are input from the analysis processing unit 184, the learning unit 186 stores the analysis results for a predetermined period. The learning unit 186 stores, for each detection target, the time-series data of the position, the time-series data of the orientation of the body, and the time-series data of the orientation of the face as one set. Then, the learning unit 186 performs learning for path prediction using a plurality of sets (a plurality of detection targets) of time-series data. The learning target is the same model as the prediction unit 188 described later. The learning algorithm is arbitrary. For example, depending on the model to be used, a neural network, a Markov decision process, a Gaussian process, etc. can be used. The learning unit 186 outputs the learning results (for example, learning parameters (hereinafter, also simply referred to as parameters)) to the prediction unit 188.

[0053] The prediction unit 188 applies the learning result input from the learning unit 186 to a predetermined prediction model, inputs the analysis result input from the analysis processing unit 184 to the prediction model (the prediction model after learning), and derives a future movement route (predicted route). The prediction unit 188 outputs the derived predicted route to the packet transmission unit 182.

[0054] The packet transmission unit 182 generates packet data including the predicted route input from the prediction unit 188 and distributes it (for example, broadcasts it) to the in-vehicle device. The in-vehicle device that has received the predicted route can use the predicted route as driving support information.

[0055] The function of the packet reception unit 180 is realized by, for example, the communication unit 124 and the memory 122 in FIG. 3. The functions of the analysis processing unit 184, the learning unit 186, and the prediction unit 188 are realized by, for example, the control unit 120 and the memory 122 in FIG. 3. The function of the packet transmission unit 182 is realized by, for example, the communication unit 124 and the memory 122 in FIG. 3. The route prediction device 102 may realize the functions of the analysis processing unit 184, the learning unit 186, and the prediction unit 188 by dedicated hardware (circuit board, ASIC, etc.).

[0056] [Operation of the Route Prediction Device] (Route Prediction Processing) With reference to FIG. 8, the route prediction processing by the route prediction device 102 will be described more specifically. The processing shown in FIG. 8 is realized by the control unit 120 reading and executing a predetermined program from the memory 122.

[0057] In step 300, the control unit 120 determines whether sensor data has been received. If it is determined that the data has been received, the control proceeds to step 302. Otherwise, step 300 is repeated.

[0058] In step 302, the control unit 120 analyzes the sensor data received in step 300 to detect a pedestrian and detect its position and signs (body orientation and face orientation). Thereafter, the control proceeds to step 304. The process of step 302 corresponds to the function of the analysis processing unit 184 in FIG. 6.

[0059] In step 304, the control unit 120 determines whether to execute route prediction. If it is determined to execute, the control proceeds to step 306. Otherwise, the control proceeds to step 310. For example, if as a result of the analysis in step 302, a pedestrian is detected and its position and signs (body orientation and face orientation) can be detected, the determination result is YES. If as a result of the analysis in step 302, no pedestrian is detected, the determination result is NO. If as a result of the analysis in step 302, a pedestrian is detected but its position and signs cannot be detected, the determination result is NO.

[0060] In step 306, the control unit 120 uses the position and signs detected in step 302 to predict the movement route for each detected pedestrian. Thereafter, the control proceeds to step 308. The process of step 306 corresponds to the function of the prediction unit 188 in FIG. 6.

[0061] In step 308, the control unit 120 broadcasts the predicted route generated in step 306. Thereafter, the control proceeds to step 310.

[0062] In step 310, the control unit 120 determines whether an end instruction has been received. If it is determined that an end instruction has been received, this program ends. Otherwise, the control returns to step 300. The end instruction is made, for example, when the route prediction device 102 is operated by an administrator or the like.

[0063] As a result, the predicted route transmitted from the route prediction device 102 is received by the in-vehicle device. The in-vehicle device that has received the predicted route can use the predicted route as driving support information. For example, the in-vehicle device can determine whether the traveling direction of the host vehicle intersects the predicted route, and if they intersect, can prompt the driver to pay attention by presenting a warning or the like.

[0064] By using the position and signs of moving objects such as detected pedestrians, the moving route of the moving object can be predicted earlier before the moving object actually acts.

[0065] The direction of the pedestrian's body does not change unless it is time for the pedestrian to change the traveling direction, so the accuracy of the predicted route based only on the direction of the body is not very high. On the other hand, usually, when an outdoor pedestrian changes the traveling direction, in order to avoid collision with other moving objects (vehicles, pedestrians, etc.), the pedestrian visually checks by turning the face toward the surroundings including the new traveling direction. Therefore, by using the direction of the face as a sign of a change in the travel route, the predicted route can be determined accurately. In particular, by using the direction of the body and the direction of the face, the predicted route can be determined more accurately.

[0066] For example, referring to FIG. 2, assume that the traffic signal (pedestrian traffic signal) 112 and the vehicle traffic signal 206 related to lateral movement are green, and the pedestrian traffic signal 204 and the vehicle traffic signal 208 related to vertical movement are red. In FIG. 2, the current traveling directions of people and vehicles are indicated by solid arrows, and the predicted future movement paths, i.e., the predicted paths, are indicated by dashed lines. The detection areas of the roadside sensor 104 and the in-vehicle sensors of the vehicle 106 are indicated by fan-shaped dashed-dotted lines. The sensor data transmitted from the roadside sensor 104 to the path prediction device 102 includes an image of the pedestrian 200. The path prediction device 102 detects the pedestrian 200 from the sensor data received from the roadside sensor 104, detects its position and signs (body orientation and face orientation), and uses them to execute path prediction, thereby determining the predicted path 210. The path prediction device 102 distributes information specifying the determined predicted path 210. As a result, the in-vehicle device of the vehicle 116 attempting to turn right that has received the information specifying the predicted path 210 can give a prompt to alert the driver because the predicted path 210 overlaps the crosswalk that the vehicle 116 is about to enter.

[0067] In addition, the sensor data transmitted from the in-vehicle device of the vehicle 106 includes an image of the pedestrian 202. The route prediction device 102 detects the pedestrian 200 from the sensor data received from the in-vehicle device of the vehicle 106, detects its position and signs (body orientation and face orientation), and uses them to execute route prediction, thereby determining the predicted route 212. The route prediction device 102 distributes information specifying the determined predicted route 212. Thereby, the in-vehicle devices of the vehicles 106 and 118 that have received the information specifying the predicted route 212 can detect that the pedestrian 202 is about to enter the roadway and give a prompt to alert the driver. Thereby, the in-vehicle device of the vehicle 118 can detect that there is a pedestrian 202 who is about to cross in front of the vehicle 118 before the driver of the vehicle 118 visually perceives it, and can alert the driver. Also, the in-vehicle device of the vehicle 106 can also alert the driver in advance, and is effective when the driver cannot recognize that the pedestrian 202 is about to cross the road even if the pedestrian 202 is within the driver's field of view of the vehicle 106.

[0068] It is possible to predict the route of a pedestrian near an intersection using only the sensor data of the roadside sensor 104. However, if the roadside sensor and the pedestrian are far apart, it may be difficult to detect the signs of the pedestrian depending on the performance of the roadside sensor 104. For example, if the resolution of the image is not sufficient, the direction of the pedestrian's face cannot be determined. Therefore, as described above, it is effective to use not only the sensor data of the roadside sensor 104 but also the sensor data of the in-vehicle device. Not only in the case described with reference to FIG. 2, but also in the case of FIG. 9, for example, it is effective to use the sensor data of the in-vehicle device. Referring to FIG. 9, the sensor data transmitted from the in-vehicle device of the vehicle 106 to the route prediction device 102 includes an image of the pedestrian 202. The route prediction device 102 detects the pedestrian 202 from the sensor data received from the in-vehicle device of the vehicle 106, detects its position and signs (body direction and face direction), and uses them to execute route prediction, thereby determining the predicted route 212. The route prediction device 102 distributes information specifying the determined predicted route 212. As a result, the in-vehicle device of the vehicle 114 that has received the information specifying the predicted route 212 can detect that the driving direction after the left turn of the vehicle 114 intersects with the predicted route of the pedestrian 202 when the vehicle 114 is about to make a left turn, and can give a prompt to alert the driver. Even after the vehicle 114 has made a left turn, the pedestrian 202 located in a blind spot due to an obstacle 240 such as a tree is not visible to the driver of the vehicle 114, so it is effective to be able to give a prior alert.

[0069] Note that not only the sensor data transmitted from the in-vehicle device of a moving vehicle but also the sensor data transmitted from the in-vehicle sensor of a parked vehicle may be used for route prediction. For example, there is a pedestrian who tries to enter and cross the road between a plurality of vehicles parked in a row. In such a case, it is preferable to use the sensor data transmitted from the in-vehicle device of the parked vehicle to detect the pedestrian and predict its movement route.

[0070] (Learning process) With reference to FIG. 10, the learning process related to the route prediction by the route prediction device 102 will be described more specifically. The process shown in FIG. 10 is realized by the control unit 120 reading and executing a predetermined program from the memory 122.

[0071] In step 400, the control unit 120 determines whether sensor data has been received. If it is determined that the data has been received, the control proceeds to step 402. Otherwise, step 400 is repeated.

[0072] In step 402, the control unit 120 analyzes the sensor data received in step 400 to detect a pedestrian and detect its position and signs (body orientation and face orientation). Thereafter, the control proceeds to step 404. The process of step 402 corresponds to the function of the analysis processing unit 184 in FIG. 6.

[0073] In step 404, the control unit 120 stores the position and signs detected in step 402 for each detected pedestrian. Thereafter, the control proceeds to step 406.

[0074] In step 406, the control unit 120 determines whether to execute learning. If it is determined to execute, the control proceeds to step 408. Otherwise, the control proceeds to step 412. For example, when the time-series data of the position and signs stored in step 404 reaches a predetermined amount or more, it is determined to execute learning. Also, it may be determined to execute learning every time a predetermined time elapses or at a predetermined time.

[0075] In step 408, the control unit 120 uses the stored time series data of positions and signs to perform learning of a prediction model to be used for path prediction. Thereafter, the control proceeds to step 410. For example, input data for learning and teacher data are automatically generated from the stored time series data of positions and signs, and the prediction model is learned using them. Specifically, referring to FIG. 11, when a path (two-dimensional path) from a starting point A via an intermediate point B to an end point C is stored as the time series data of the position of a detected pedestrian, the data from the starting point A to the intermediate point B (time series data of the position) is used as the input data. Also, the data from the intermediate point B to the end point C is used as the teacher data. Similarly, the time series data of signs (body orientation, face orientation) is also divided into input data and teacher data. In this way, for example, one input data and the corresponding teacher data are generated from the time series data of positions and signs regarding a certain pedestrian. Similarly, input data and teacher data are generated from the time series data of positions and signs regarding other pedestrians. By performing learning using the plurality of input data and teacher data generated in this way, the parameters of the prediction model are determined. The process of step 408 corresponds to the function of the learning unit 186 in FIG. 6.

[0076] In step 410, the control unit 120 stores the learning result (parameters) in step 408. Thereafter, the control proceeds to step 412.

[0077] In step 412, the control unit 120 determines whether or not an end instruction has been received. If it is determined that an end instruction has been received, this program ends. Otherwise, the control returns to step 400. The end instruction is made, for example, by the path prediction device 102 being operated by an administrator or the like.

[0078] Thereby, the learning result (parameters) stored in step 410 is used in step 306 of the above-described path prediction process. As described above, by automatically generating input data for learning and teacher data, learning can be efficiently executed.

[0079] In the above description, the case where the route prediction device 102 and the roadside sensor 104 are arranged with an intersection having a crosswalk as an observation target has been described, but it is not limited thereto. The route prediction device 102 and the roadside sensor 104 can be arranged at any location around a road where vehicles pass, to detect pedestrians and perform route prediction for them. For example, even if it is not on a crosswalk, by installing the route prediction device 102 and the roadside sensor 104 for a location where pedestrians tend to cross, a large amount of route data can be obtained, so the accuracy of route prediction can be improved.

[0080] Also, when only the route prediction device 102 is arranged without arranging the roadside sensor 104, pedestrians can be detected and their route prediction can be performed by using the sensor data of the sensors mounted on the vehicle.

[0081] In the above description, the case of performing route prediction using only sensor data has been described, but it is not limited thereto. The information of traffic signals may be used for route prediction. For example, if the pedestrian signal is red, it is generally considered that the possibility of a movement route such as crossing the crosswalk at that time is low. Therefore, by using the information of traffic signals (including pedestrian signals and vehicle signals) for route prediction, the accuracy of route prediction can be improved.

[0082] The sensor data is not limited to the data imaged by a camera. The sensor may be a detection device that outputs data capable of detecting pedestrians by analysis and calculating their feature amounts (position and signs), and may be a device that repeatedly captures still images at short intervals, a sensor using RF (such as millimeter waves) or laser.

[0083] In the above description, pedestrians are the detection target, but it is not limited thereto. Moving objects that may cause a vehicle to collide and cause damage can be the detection target, and it may be a person riding a bicycle.

[0084] Since the vehicle is moving, the range within which the in-vehicle sensor can acquire sensor data is different from the range within which the roadside sensor can acquire sensor data. The ranges within which in-vehicle sensors of different vehicles can acquire sensor data are also different. On the other hand, the same pedestrian may be included in the sensor data acquired by the roadside sensor and the in-vehicle sensor. Therefore, in order to detect pedestrians over a wider range and determine their predicted paths, it is preferable to fuse and use the sensor data of the roadside sensor and the in-vehicle sensor, and further the detection results of the sensor data of a plurality of in-vehicle sensors. As a result, more information can be obtained regarding the same pedestrian detected over a wider range, and their position and signs can be detected with higher accuracy. Therefore, the predicted path can be determined with higher accuracy.

[0085] In the above, the case where the path prediction device 102 distributes only the predicted path has been described, but it is not limited to this. In addition to the predicted path, auxiliary data may be distributed. For example, the detected image data of the moving object corresponding to the predicted path may be transmitted. Thereby, the in-vehicle device can visually present the image of the moving object such as a pedestrian who is the target of the predicted path to the driver, so that the reliability of the received path can be confirmed.

[0086] In the above, the case where the position, body orientation, and face orientation of the detected pedestrian are used for path prediction has been described, but it is not limited to this. In addition to these, for example, the moving speed of the detected pedestrian may be used. Also, instead of the body orientation and face orientation, or in addition to at least one of the body orientation and face orientation, another body state of the pedestrian that can be observed before changing the traveling direction may be used as a sign.

[0087] In the above, the case where the fixedly installed path prediction device 102 performs path prediction using the sensor data received from the roadside sensor and the in-vehicle device has been described, but it is not limited to this. The path prediction device may be mounted on a vehicle. For example, the in-vehicle device may perform learning for path prediction and path prediction.

[0088] (Second Embodiment) The prediction device (prediction model) after learning for route prediction may use the same one on roads across the country, but the prediction accuracy depends on the past data used for learning. That is, it is considered that the sensor data used for learning reflects the geographical conditions (road shape, lane width, number of lanes, presence or absence of sidewalks, sidewalk width, presence or absence of traffic signals, position of traffic signals, traffic volume, etc.) of the area (hereinafter also referred to as area) where the sensor data was acquired. It is preferable that a route prediction device that performs route prediction in a specific area is learned by the sensor data acquired in that area. Therefore, in the second embodiment, the in-vehicle device has a route prediction function, and uses the learning result determined by learning for each area to perform route prediction in the area where the vehicle is traveling.

[0089] Referring to FIG. 12, the route prediction system according to the second embodiment includes a server 500 arranged in area 510, a server 502 arranged in area 512 different from area 510, and in-vehicle devices of a plurality of vehicles 504 and 508. Servers 500 and 502 are computers and are configured in the same manner as in FIG. 3. In the memory of server 500, learning parameters determined by learning the prediction model as described above using sensor data from roadside sensors (not shown) arranged in area 510 are stored. Similarly, in the memory of server 502, learning parameters determined using sensor data from roadside sensors (not shown) arranged in area 512 are stored. Each of servers 500 and 502 appropriately distributes the learning parameters stored in its own memory and functions as a parameter distribution device. In FIG. 12, two areas 510 and 512 are illustrated, but it is assumed that a predetermined area including roads (for example, the whole country, each prefecture, etc.) is divided into a plurality of areas, and at least one server (storing and distributing learning parameters) is arranged in each area.

[0090] (Functional configuration of in-vehicle device) The in-vehicle devices of vehicles 504 and 506 are configured in the same way as in FIG. 4. The functional configuration of the in-vehicle devices of vehicles 504 and 506 is shown in FIG. 13. Referring to FIG. 13, the in-vehicle device 520 includes a packet reception unit 522, a parameter storage unit 524, a sensor unit 526, an analysis processing unit 528, a prediction unit 530, and an information presentation unit 532.

[0091] The packet reception unit 522 receives packet data (learning parameters transmitted from servers 500 and 502), and outputs the received learning parameters to the parameter storage unit 524. The parameter storage unit 524 stores in advance learning parameters corresponding to each area. The parameter storage unit 524 appropriately stores the learning parameters input from the packet reception unit 522.

[0092] The sensor unit 526 is configured in the same way as the sensor unit 142 and the I / F unit 144 in FIG. 4, for example, and acquires sensor data. The acquired sensor data is output to the analysis processing unit 528. The analysis processing unit 528 analyzes the input sensor data in the same way as the analysis processing unit 184 in FIG. 6, detects a pedestrian, and detects the position and signs (the orientation of the pedestrian's body and the orientation of the face). The analysis processing unit 528 outputs the detected position and signs to the prediction unit 530.

[0093] When the position and signs are input from the analysis processing unit 528, the prediction unit 530 reads out the learning parameters of the area where the vehicle is currently traveling (the area where the vehicle is currently located) from the learning parameters stored in the parameter storage unit 524, and executes route prediction in the same way as the prediction unit 188 in FIG. 6. The prediction unit 530 outputs the obtained route prediction to the information presentation unit 532.

[0094] Referring to FIG. 14, an example of learning parameters will be described. FIG. 14 shows a predictor (prediction model) using a neural network. Measured data is input to each node in the input layer. When data is input to a node in a certain layer, that node multiplies the input value by a predetermined weight and outputs the resulting value to each node in the next layer. By repeating this between adjacent layers, data is output from the output layer. When the weights set for the nodes in each layer are changed, the output data of the output layer changes. In learning, using a large amount of input data, each weight is adjusted so that the output data for each input data approaches the teacher data. In the case of such a neural network model, the weights are the learning parameters.

[0095] In advance, such a neural network model is learned using input data and teacher data generated from sensor data acquired in each area, and the resulting weights are stored as learning parameters in the server arranged in that area. The input data and teacher data can be automatically generated, for example, from time-series data of the position and signs (body orientation and face orientation) of pedestrians detected from the sensor data as shown in FIG. 11.

[0096] In FIG. 14, the case where there are two intermediate layers is shown, but the intermediate layer may be one or three or more. Also, the prediction model is not limited to a neural network. Any prediction model in which the learning parameters are determined by learning using the position and signs (body orientation and face orientation) of pedestrians detected from the sensor data is acceptable. The prediction unit 530 of the in-vehicle device 520 may adopt a prediction model having the same configuration as the prediction model that determined the learning parameters.

[0097] Returning to FIG. 13, the information presentation unit 532 corresponds to the bus 154 in FIG. 4 and includes, for example, an image display device such as a liquid crystal display and an acoustic device such as a speaker, and displays the route prediction (for example, a series of two-dimensional coordinates) input from the prediction unit 530 as an image. Also, it determines whether the route prediction intersects the traveling direction of the vehicle, and if it is determined that they intersect, it emits a warning sound to prompt attention.

[0098] (Operation of In-vehicle Device) Referring to FIG. 15, the route prediction process by the in-vehicle device 520 will be described in more detail. As described above, since the in-vehicle device 520 is configured in the same manner as in FIG. 4, the reference numerals in FIG. 4 will be referred to below. The process shown in FIG. 15 is realized by the control unit 150 reading and executing a predetermined program from the memory 148. Here, it is assumed that the memory 148 stores information for specifying an area in advance (such as map information) and learning parameters corresponding thereto.

[0099] In step 600, the control unit 150 specifies the area where the vehicle is currently traveling. Specifically, the control unit 150 acquires the current position of the vehicle by means of GPS or the like, and specifies the area in which the position is included from the area information stored in the memory 148. The control unit 150 stores, in a predetermined area of the memory 148, information representing the specified area (hereinafter referred to as the traveling area information).

[0100] In step 602, the control unit 150 determines whether the traveling area of the vehicle has changed. Specifically, the control unit 150, in the same manner as in step 600, specifies the area where the vehicle is currently traveling, and determines whether the specified area is different from the area specified by the traveling area information stored in the memory 148. If it is determined that they are different (the area has changed), after overwriting the information representing the specified area on the traveling area information in the memory 148, the area control proceeds to step 604. Otherwise, the control proceeds to step 616.

[0101] In step 604, the control unit 150 determines whether the memory 148 stores learning parameters corresponding to the area specified in step 602. If it is determined that they are stored, the control proceeds to step 606. Otherwise, the control proceeds to step 612.

[0102] In step 606, the control unit 150 queries the server of the area where the vehicle is traveling about the version of the learning parameters. Specifically, the control unit 150 broadcasts a predetermined code (hereinafter referred to as the version query code). As will be described later, the server that has received the version query code transmits information (version information) representing the version of the learning parameters stored therein to the in-vehicle device that has transmitted the version query code. For example, when the in-vehicle device of the vehicle 504 traveling in the area 510 broadcasts the version query code, the server 500 transmits the version information to the vehicle 504. The in-vehicle devices of the vehicles traveling around the vehicle 504 ignore the received version query code.

[0103] In step 608, the control unit 150 determines whether the version information has been received. If it is determined that the version information has been received, the control proceeds to step 610. Otherwise, the process of step 608 is repeated.

[0104] In step 610, the control unit 150 determines whether the version information obtained in step 608 is newer than the version of the learning parameters corresponding to the area specified in step 602 among the learning parameters stored in the memory 148. If it is determined that the version information is newer (i.e., if the learning parameters stored in the memory 148 are old), the control proceeds to step 612. Otherwise (i.e., if the learning parameters stored in the memory 148 are new or equivalent), the control proceeds to step 616.

[0105] In step 612, the control unit 150 requests the server in the area where the vehicle is traveling to transmit the learning parameters. Specifically, the control unit 150 broadcasts a predetermined code (hereinafter referred to as a transmission request code). As will be described later, the server that has received the transmission request code transmits the stored learning parameters to the in-vehicle device that has transmitted the transmission request code. For example, when the in-vehicle device of the vehicle 504 traveling in area 510 broadcasts the transmission request code, the server 500 transmits the learning parameters to the vehicle 504. The in-vehicle devices of the vehicles traveling around the vehicle 504 ignore the received transmission request code.

[0106] In step 614, the control unit 150 determines whether the learning parameters have been received. If it is determined that they have been received, the received learning parameters are stored in the memory 148 in association with the area specified in step 620. Thereafter, the control proceeds to step 616. Otherwise, step 614 is repeated.

[0107] In step 616, the control unit 150 determines whether the sensor data has been updated, that is, whether new sensor data has been acquired from the sensor unit 142. If it is determined that it has been received, the control proceeds to step 618. Otherwise, the control proceeds to step 624.

[0108] In step 618, the control unit 150 analyzes the sensor data acquired in step 616 (corresponding to the function of the analysis processing unit 528 in FIG. 13) and determines whether the detection target (pedestrian) has been detected and its position and signs have been detected. If it is determined that they have been detected, the control proceeds to step 620. Otherwise (for example, if the detection target has not been detected, or if the detection target has been detected but either its position or signs have not been detected), the control proceeds to step 624.

[0109] In step 620, the control unit 150 reads out the learning parameters corresponding to the area specified in step 602, and uses the read learning parameters to predict the movement path of the pedestrian detected in step 616 from the position and signs (body orientation and face orientation) detected in step 616 (corresponding to the function of the prediction unit 530 in FIG. 13), and stores the resulting predicted path in the memory 148.

[0110] In step 622, the control unit 150 reads out the predicted path, which is the prediction result of step 620, from the memory 148 and presents it to the presentation unit 152 (function of the information presentation unit 532 in FIG. 13).

[0111] In step 624, the control unit 150 determines whether an end instruction has been received. If it is determined that an end instruction has been received, this program ends. Otherwise, the control returns to step 602. The end instruction is made, for example, by turning off the engine and electrical system of the vehicle on which the in-vehicle device is mounted.

[0112] (Server operation) With reference to FIG. 16, the processing by the servers 500 and 502 will be described more specifically. As described above, since the servers 500 and 502 are configured in the same manner as in FIG. 3, the reference numerals in FIG. 3 will be referred to hereinafter. The processing shown in FIG. 16 is realized by the control unit 120 reading out and executing a predetermined program from the memory 122. The memory 122 stores the learning parameters of the corresponding area and their version information. The learning parameters stored in the server 500 are different from the learning parameters stored in the server 502.

[0113] In step 700, the control unit 120 determines whether there is an inquiry about the version of the learning parameters. Specifically, the control unit 120 determines whether it has received a version inquiry code. If it is determined that the code has been received, the control proceeds to step 702. Otherwise, the control proceeds to step 708. As described above, the version inquiry code is transmitted from the in-vehicle device (see step 606 in FIG. 15).

[0114] In step 702, the control unit 120 transmits the version information of the learning parameters stored in the memory 122 to the in-vehicle device that sent the version inquiry code. Then, the control proceeds to step 704. Information (address) for identifying the in-vehicle device that sent the version inquiry code can be obtained from the source address of the packet including the version inquiry code received in step 700. The transmitted version information is received by the in-vehicle device that sent the version inquiry code (see step 608 in FIG. 15).

[0115] In step 704, the control unit 120 determines whether there is a request for learning parameters. Specifically, the control unit 120 determines whether it has received a transmission request code. If it is determined that the code has been received, the control proceeds to step 706. Otherwise, the control proceeds to step 708. As described above, the transmission request code is transmitted from the in-vehicle device (see step 612 in FIG. 15).

[0116] In step 706, the control unit 120 reads out the learning parameters from the memory 122 and transmits them to the in-vehicle device that sent the transmission request code. Then, the control proceeds to step 708. Information (address) for identifying the in-vehicle device that sent the transmission request code can be obtained from the source address of the packet including the transmission request code received in step 704. The transmitted learning parameters are received by the in-vehicle device that sent the transmission request code (see step 614 in FIG. 15).

[0117] In step 708, the control unit 120 determines whether an end instruction has been received. If it is determined that an end instruction has been received, this program ends. Otherwise, the control returns to step 700. The end instruction is made, for example, by the server including the control unit 120 being operated by an administrator or the like.

[0118] As described above, if the in-vehicle device stores the learning parameters corresponding to the area during the vehicle's travel, it can use them to determine the predicted path of the detected pedestrian. On the other hand, if the in-vehicle device does not store the learning parameters corresponding to the area during the vehicle's travel, or if the stored learning parameters are older than the learning parameters that can be obtained from the server, the in-vehicle device can obtain the learning parameters from the server and use them to determine the predicted path of the detected pedestrian.

[0119] For example, referring to FIG. 12, the in-vehicle device of the vehicle 504 traveling in the area 510 obtains, as necessary, the learning parameters suitable for the area 510 from the server 500 and performs path prediction using them. The in-vehicle device of the vehicle 504 analyzes the sensor data of the sensors mounted on the vehicle 504. For example, when a pedestrian 590 is detected, the predicted path 592 of the pedestrian 590 can be determined from its position and signs using the learning parameters received from the server 500. Also, when the vehicle 504 comes to travel in the area 512, the in-vehicle device of the vehicle 504 can obtain, as necessary, the learning parameters suitable for the area 512 from the server 502. The in-vehicle device of the vehicle 504 analyzes the sensor data of the sensors mounted on the vehicle 504. For example, when a pedestrian 594 is detected, the predicted path 596 of the pedestrian 594 can be determined from its position and signs using the learning parameters received from the server 502.

[0120] In this way, the in-vehicle device can more accurately determine the predicted path of the detected pedestrian by using the learning parameters suitable for the area where the mounted vehicle is traveling. Therefore, the in-vehicle device can provide more appropriate driving support to the driver, such as prompting attention based on the highly accurate predicted path.

[0121] Only when the server receives a request from the in-vehicle device, by transmitting the learning parameters to the requested in-vehicle device, it is possible to suppress the transmission of learning parameters unnecessary for the in-vehicle device from the server.

[0122] In the above, the case where the in-vehicle device stores in advance the learning parameters for each of a plurality of areas in a predetermined region has been described. Thereby, the in-vehicle device alone can accurately predict the movement route of a moving object such as a pedestrian earlier, using appropriate predetermined information (parameters) according to the region, before the moving object actually moves.

[0123] On the other hand, by enabling the in-vehicle device to acquire learning parameters from the servers arranged in each area as needed, the in-vehicle device does not have to store the learning parameters in advance, and can suppress storing unnecessary data.

[0124] Although the present invention has been described by explaining the embodiments above, the above-described embodiments are examples, and the present invention is not limited only to the above-described embodiments. The scope of the present invention is indicated by each claim of the claims, taking into consideration the description of the detailed description of the invention, and includes all changes within the meaning and scope equivalent to the language described therein.

Explanation of Signs

[0125] 100 Route prediction system 102 Route prediction device 104 Roadside sensor 106, 114, 116, 118, 504, 506 Vehicles 108 Base station 110 Network 112 Traffic signal (pedestrian traffic signal) 120, 150, 168 Control unit 122, 148, 166 Memory 124, 146, 164 Communication unit 126, 154, 170 Bus 140, 520 In-vehicle device Sensor units 142, 160, 526 I / F units 144, 162 Presentation unit 152 Packet reception units 180, 522 Packet transmission unit 182 Analysis processing units 184, 528 Learning unit 186 Prediction units 188, 530 Pedestrians 200, 202, 590, 594 Traffic signal for pedestrians 204 Traffic signals for vehicles 206, 208 Predicted routes 210, 212, 592, 596 Target frames 220, 230 Face area frame 222 Face direction 224 Body area frame 226 Body direction 228 Obstacle 240 Servers 500, 502 Areas 510, 512 Parameter storage unit 524 Information presentation unit 532

Claims

1. A collecting unit that collects sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication; An analysis unit that detects a person from the sensor data collected by the collecting unit and specifies a sign including information representing the position of the person and the direction of the person's face; A prediction unit that predicts a route including the future position of the person using the position and the sign of the person specified by the analysis unit; A storage unit that stores the position and the sign specified by the analysis unit in correspondence with each person; A learning unit that uses the position and the sign stored in the storage unit for a predetermined period to train the prediction unit; A route prediction device, comprising: an evaluation unit that evaluates the possibility of collision between the person for whom the route is predicted by the prediction unit and a moving object different from the person using the route predicted by the prediction unit.

2. The route prediction device according to claim 1, wherein the sign further includes information representing the orientation of the person's body.

3. The learning unit: Trains the prediction unit by machine learning, Of the time-series data of the position and the sign that are specified for each person over a predetermined time and stored in the storage unit, the first time-series data of the position and the sign belonging to the more past is used as input data for the machine learning, The route prediction device according to claim 1 or claim 2, wherein time-series data that is specified for each person over a predetermined time and stored in the storage unit and does not include the first time-series data is used as teacher data for the machine learning.

4. The route prediction device according to any one of claims 1 to 3, further comprising a distribution unit that distributes the route predicted by the prediction unit to an in-vehicle device.

5. The distribution unit distributes the image data of the person corresponding to the route together with the route predicted by the prediction unit, the route prediction device according to claim 4.

6. The collection unit collects the sensor data acquired by the in-vehicle sensor and the sensor data acquired by the roadside sensor, further includes a determination unit that determines whether or not a first person detected from the sensor data acquired by the in-vehicle sensor and a second person detected from the sensor data acquired by the roadside sensor are the same person, The prediction unit predicts the route using both the position and the sign specified for each of the first person and the second person determined to be the same person, the route prediction device according to any one of claims 1 to 5.

7. An in-vehicle device including the route prediction device according to any one of claims 1 to 6.

8. An in-vehicle device mounted on a vehicle including a first sensor, A roadside device including a second sensor installed on the road, including a route prediction device, The in-vehicle device transmits the sensor data acquired by the first sensor to the route prediction device via wireless communication, The roadside device transmits the sensor data acquired by the second sensor to the route prediction device via wireless communication or wired communication, The route prediction device, a receiving unit that receives the sensor data transmitted from the in-vehicle device and the sensor data transmitted from the roadside device, an analysis unit that detects a person from the sensor data received by the receiving unit and specifies a sign including information representing the position of the person and the orientation of the face of the person, a prediction unit that predicts a route including the future position of the person using the position and the sign of the person specified by the analysis unit, A storage unit that associates and stores the position and the sign specified by the analysis unit for each person; A learning unit that trains the prediction unit using the position and the sign stored in the storage unit for a predetermined period; An evaluation unit that evaluates the possibility of collision between the person who is the prediction target of the route predicted by the prediction unit and another moving object using the route; a route prediction system including.

9. A collection step of collecting sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication; An analysis step of detecting a person from the sensor data collected in the collection step and specifying a sign including information representing the position of the person and the direction of the face of the person; A prediction step of predicting a route including the future position of the person using the position and the sign of the person specified in the analysis step; A storage step of associating and storing the position and the sign specified in the analysis step for each person; A learning step of training the prediction step using the position and the sign stored for a predetermined period in the storage step; An evaluation step of evaluating the possibility of collision between the person who is the prediction target of the route predicted in the prediction step and another moving object using the route; a route prediction method including.

10. In a computer, A collection function of collecting sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication; An analysis function of detecting a person from the sensor data collected by the collection function and specifying a sign including information representing the position of the person and the direction of the face of the person; A prediction function of predicting a route including the future position of the person using the position and the sign of the person specified by the analysis function; A storage function that associates and stores the position and the sign identified by the analysis function for each person; A learning function that learns the prediction function using the position and the sign stored by the storage function for a predetermined period; A computer program that realizes an evaluation function that evaluates the possibility of a collision between the person to be predicted on the route and another moving object using the route predicted by the prediction function.

Images