Moving body behavior prediction device, vehicle control device, and moving body behavior prediction method

Abstract

This moving body behavior prediction device comprises a model storage unit and a moving body behavior prediction unit. The model storage unit receives as input: time series information including position information of a host vehicle and a moving body around the host vehicle at different times; and host vehicle state information that is information indicating an operation or a state of the host vehicle displayed by the host vehicle or an occupant of the host vehicle, and that can be recognized by the moving body, and stores a moving body behavior prediction model that outputs a behavior prediction result of the moving body around the host vehicle. The moving body behavior prediction unit inputs the acquired time series information and host vehicle state information displayed by the host vehicle or the occupant of the host vehicle to the moving body behavior prediction model, and outputs a behavior prediction result of the moving body.

Description

Mobile object behavior prediction device, vehicle control device, and mobile object behavior prediction method

[0001] The present invention relates to a mobile object behavior prediction device, a vehicle control device, and a mobile object behavior prediction method for predicting the future behavior of a mobile object such as a vehicle or a pedestrian.

[0002] Various technologies have been proposed for detecting objects (vehicles, motorcycles, bicycles, pedestrians, structures, etc.) around a vehicle using external recognition sensors such as on-board cameras and radar. Furthermore, mobile object behavior prediction technologies have been developed that use these technologies to predict the future behavior of detected objects, particularly moving objects such as vehicles and pedestrians. Such technologies are necessary for achieving advanced driver assistance and autonomous driving, and methods that utilize machine learning to adapt to diverse environments are also being actively developed.

[0003] Patent Literature 1 discloses an example of such a moving object behavior prediction technology. Patent Literature 1 discloses a vehicle driving assistance device that detects a moving obstacle around the vehicle, calculates the moving obstacle's path of travel, and then provides driving assistance based on the calculated path of travel. The path of travel of the moving obstacle is predicted taking into consideration factors such as the degree of proximity to obstacles other than the moving obstacle. Therefore, this vehicle driving assistance device can predict an appropriate path of travel and provide appropriate driving assistance even in situations where the moving obstacle's motion state is expected to change due to interference with other obstacles.

[0004] JP 2009-116790 A

[0005] In situations where vehicles and other moving objects are approaching each other, such as on narrow public roads near residential areas or in parking lots, vehicles may give way to other moving objects. In such situations, it is necessary to predict the behavior of moving objects taking into account the interactions between the vehicle and surrounding moving objects. For example, a common driving method is to drive smoothly while expressing each other's intentions by using turn signals, flashing lights, etc.

[0006] The technology disclosed in Patent Document 1 predicts the behavior (path of travel) of a target moving obstacle based on information about the obstacle's position and speed, but it is difficult to predict the behavior of the moving obstacle taking into account the interaction between the vehicle and the moving obstacle.

[0007] The present invention has been made in consideration of the above-mentioned circumstances, and has an object to make it possible to predict with high accuracy the behavior of a moving object that interacts with the host vehicle.

[0008] In order to solve the above problems, a mobile object behavior prediction device according to one aspect of the present invention includes a model storage unit and a mobile object behavior prediction unit. The model storage unit stores a mobile object behavior prediction model that receives as input time-series information including position information of the host vehicle and mobile objects around the host vehicle at different times and host vehicle state information that is information that represents the operation or state of the host vehicle and is displayed by the host vehicle or a passenger of the host vehicle and is recognizable by the mobile object, and outputs a behavior prediction result of the mobile object around the host vehicle. The mobile object behavior prediction unit inputs the time-series information including position information of the host vehicle and mobile objects around the host vehicle acquired at different times and the host vehicle state information that is information that represents the operation or state of the host vehicle and is displayed by the host vehicle or a passenger of the host vehicle and is recognizable by the mobile object, to the mobile object behavior prediction model, and outputs a behavior prediction result of the mobile object.

[0009] According to at least one aspect of the present invention, it is possible to predict with high accuracy the behavior of a moving object that interacts with the host vehicle. Problems, configurations, and effects other than those described above will become apparent from the following description of the preferred embodiment of the present invention.

[0010] FIG. 1 is a schematic configuration diagram showing an example of a vehicle control device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the hardware configuration of a computer provided in the vehicle control device according to an embodiment of the present invention. FIG. 3 is a schematic configuration diagram showing a first example of a moving object behavior prediction device in the vehicle control device according to an embodiment of the present invention. FIG. 4 is a schematic configuration diagram showing a second example of a moving object behavior prediction device in the vehicle control device according to an embodiment of the present invention. FIG. 5 is a flowchart showing an example of the procedure of the overall processing of the vehicle control device according to an embodiment of the present invention. FIG. 6 is a diagram showing an example of input information and output information in moving object behavior prediction processing. FIG. 7 is a diagram showing a first example of moving object behavior prediction processing. FIG. 8 is a diagram showing a second example of moving object behavior prediction processing. FIG. 9 is a diagram showing a third example of moving object behavior prediction processing. FIG. 10 is a diagram showing a fourth example of moving object behavior prediction processing. FIG.

[0011] Hereinafter, examples of modes for carrying out the present invention (hereinafter referred to as "embodiments") will be described with reference to the accompanying drawings. In this specification and the accompanying drawings, identical or similar components are given the same reference numerals, and redundant explanations may be omitted or only explanations focusing on the differences may be given. Furthermore, when there are multiple identical or similar components, they may be described using the same reference numerals with different subscripts. Note that when it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description. The number of each component may be singular or plural unless otherwise specified.

[0012] <One embodiment> First, a vehicle control device according to one embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a schematic configuration diagram showing an example of a vehicle control device according to one embodiment of the present invention. The vehicle control device 10 illustrated in Fig. 1 is a computer that is mounted on a vehicle and controls the vehicle. The vehicle control device 10 executes programs recorded on a recording medium (not shown) to function as a surrounding environment recognition unit 11, a map information storage unit 12, a mobile object behavior prediction unit 13, a driving assistance determination unit 14, a vehicle control unit 15, and an information notification unit 16.

[0013] The vehicle control device 10 is connected to a steering device 51, a drive device 52, a braking device 53, and a transmission device 54 of the vehicle. The vehicle control device 10 is also connected to an external environment recognition device 1, a sound generating device 55, and a display device 56 provided in the vehicle. The vehicle control device 10 is further connected to a transmission path such as a controller area network (CAN) (not shown) of the vehicle or a dedicated line. Vehicle information such as the vehicle speed, steering angle, yaw rate, lighting device status, display device status, and driver monitoring system (DMS) information is input to the vehicle control device 10 via these transmission paths.

[0014] Vehicle speed, steering angle, and yaw rate are examples of "time-series information" and are detected periodically or irregularly by various sensors (not shown) installed in the vehicle. In this specification, among these vehicle information, information that can be recognized from the moving body side is referred to as "host vehicle status information." The host vehicle status information includes at least one of the host vehicle's lighting status, indicator status, and DMS information (behavior of the occupants of the host vehicle).

[0015] Here, the lighting device status refers to the lighting status of turn signals (including hazard lights), headlights (including rapid flashing flashers, etc.), brake lights, red rotating lights (for example, in police vehicles and ambulances), etc. The display status refers to the status of the content displayed on external displays installed inside or outside the vehicle, and the status of content displayed by floodlights on walls, the ground, etc. outside the vehicle. The DMS information is information on the driver's gestures detected by an in-vehicle camera, and may also be information indicating the intention behind the gestures.

[0016] The external environment recognition device 1 is a device that acquires information about the environment surrounding the vehicle, and is, for example, an on-board camera such as a monocular camera or a stereo camera that captures the environment surrounding the vehicle. Images acquired by the on-board camera are output to the vehicle control device 10 via a transmission path such as a dedicated line, either as analog data or after A / D conversion. In addition to on-board cameras, other devices that can be used include radar that measures the distance to an object using millimeter waves or laser light, and sonar that measures the distance to an object using ultrasound. These devices, such as on-board cameras, may be configured to output information such as the distance to the detected object, its direction, speed, and type of object to the vehicle control device 10 via a transmission path such as a dedicated line.

[0017] The steering device 51 is composed of an electric power steering, a hydraulic power steering, or the like that can control the steering angle using an electric or hydraulic actuator in response to an external drive command. The drive device 52 is composed of an engine system that can control engine torque using an electric throttle or the like in response to an external drive command, and an electric powertrain system that can control the driving force of a motor or the like in response to an external drive command. The braking device 53 is composed of an electric brake, a hydraulic brake, or the like that can control braking force using an electric or hydraulic actuator in response to an external braking command. The transmission 54 is composed of a transmission that can switch between forward and reverse using an electric or hydraulic actuator in response to an external gear change command.

[0018] The sound generating device 55 is composed of a speaker or the like, and functions as an information notification unit that outputs warnings, audio guidance, etc. to the driver. The display device 56 is composed of a display of a navigation device or the like, a meter panel, warning lights, etc. The display device 56 functions as an information notification unit that notifies the driver of information such as an operation screen of the vehicle control device 10 and a warning screen. The warning screen visually conveys information to the driver that there is a risk of the vehicle colliding with an obstacle.

[0019] The surrounding environment recognition unit 11 detects the type and position of obstacles around the vehicle, as well as the direction and speed of movement of the obstacles, using image data and ranging information captured around the vehicle input from the external environment recognition device 1. Obstacles can be broadly divided into moving objects and stationary objects. Types of moving objects include vehicles, motorcycles, bicycles, and pedestrians. Types of stationary objects include guardrails, curbs, plants, parked vehicles, and poles. The type and position of obstacles are preferably detected by pattern matching, but other techniques may also be used.

[0020] The map information storage unit 12 stores map information data (map data) for various applications. For example, in the case of a conventional navigation device, road information necessary for route search and route guidance, as well as information on convenience stores, gas stations, various facilities, and the like, are stored. Furthermore, in the case of an autonomous driving application or an advanced driving assistance application, detailed information such as shape data that resembles the actual road shape represented by polygons, polylines, etc., traffic regulation information (speed limits, types of vehicles allowed on the road, etc.), lane divisions (main lane, passing lane, climbing lane, straight lane, left turn lane, right turn lane, etc.), and the presence or absence of traffic lights, signs, etc. (if present, their location information) is stored. An occupancy grid map (OGM) may be used as the map information.

[0021] The moving object behavior prediction unit 13 inputs information (time-series information) such as the position and speed of the moving object recognized by the host vehicle and surrounding environment recognition unit 11, map information acquired from the map information storage unit 12, and host vehicle status information from the vehicle information, and predicts the future behavior of the moving object. As described above, the host vehicle status information is information that can be recognized from the moving object side, and includes the status of the host vehicle's lighting devices and indicators and DMS information. Note that the behavior of moving objects interacting with the host vehicle may be predicted without using map information.

[0022] Here, it can be said that the moving object behavior prediction unit 13 predicts the behavioral intention of the own vehicle as the future behavior of the own vehicle from the own vehicle state information. Examples of behavioral intentions include turning right / left / going straight at an intersection, yielding to the other vehicle, changing lanes to the right / left, parking, leaving a parking space, stopping, starting, and remaining stopped in place. The behavioral intention of the own vehicle significantly affects the behavior of moving objects around the own vehicle that interact with the own vehicle. For example, by using information from a turn signal, which is one of the vehicle's lighting devices, and map information, etc., the moving objects around the own vehicle can determine the behavioral intention of the own vehicle, such as turning right / left or changing lanes. The moving objects around the own vehicle are likely to determine their own behavior based on the behavioral intention of the own vehicle. In this embodiment, the behavior of the moving objects around the own vehicle is predicted taking into account the influence of the behavioral intention of the own vehicle on the behavior of the moving objects around the own vehicle.

[0023] For example, the mobile object behavior prediction unit 13 predicts a sequence of trajectory points that the mobile object will pass through in the future as the future behavior of the mobile object. Each point in the trajectory point sequence has position coordinates and information on the time at that time, and the trajectory is represented by connecting each point in a time series. Alternatively, each point in the trajectory point sequence has position coordinates and information on the speed at that time, and the trajectory is represented by connecting each point in a time series. Furthermore, multiple trajectory point sequences may be predicted, and in this case, it is desirable to add information such as reliability to each trajectory point sequence. Reliability represents the likelihood (degree of reliability) of the target trajectory point sequence using an index such as probability, and can also be referred to as the accuracy of the prediction result. Note that the prediction method may be a method using machine learning, a method for searching for an optimal trajectory, or a method using rule-based conditional branching.

[0024] When machine learning is used as a method for predicting the trajectory point sequence, the mobile object behavior prediction unit 13 includes a model storage unit (not shown), and stores a mobile object behavior prediction model 131 learned using machine learning in the model storage unit. The mobile object behavior prediction unit 13 holds each piece of information input to the mobile object behavior prediction unit 13 as time-series information from the past, and inputs at least this time-series information into the mobile object behavior prediction model 131 to predict the behavior of the mobile object. The machine learning is assumed to be supervised learning. For example, an encoder / decoder model using a known long short-term memory (LSTM) or a transformer can be used as the mobile object behavior prediction model 131 using machine learning.

[0025] The driving assistance determination unit 14 inputs information about obstacles around the vehicle recognized by the surrounding environment recognition unit 11, information about the future behavior of the moving object predicted by the moving object behavior prediction unit 13, map information acquired from the map information storage unit 12, and vehicle information, and determines driving assistance required for the vehicle. Examples of driving assistance include an assistance function for avoiding collision with an obstacle, and an assistance function for realizing cooperative driving with a moving object, such as passing each other on narrow roads, and the driving assistance may also include an autonomous driving function.

[0026] The vehicle control device 10 realizes a driving assistance function by notifying the occupant of information and controlling the vehicle itself. For example, when notifying the occupant of information to realize the assistance function for avoiding a collision with an obstacle, the vehicle control device 10 calculates the approaching direction of the obstacle and the time to collision (TTC) with the obstacle. For example, when controlling the vehicle itself to realize the assistance function for avoiding a collision with an obstacle, the vehicle control device 10 calculates a target trajectory (a target route and a target speed on the target route) that will avoid a collision with the obstacle.

[0027] The vehicle control unit 15 controls the traveling of the host vehicle in accordance with the assistance content determined by the driving assistance determination unit 14. For example, when a target trajectory is generated by the driving assistance determination unit 14, the vehicle control unit 15 calculates a target steering angle and a target speed based on the target trajectory. The vehicle control unit 15 then outputs a target steering torque for achieving the target steering angle to the steering device 51. The vehicle control unit 15 also outputs a target engine torque and a target brake pressure for achieving the target speed to the drive device 52 and the braking device 53. Furthermore, when it is necessary to change the traveling direction (forward or reverse) of the host vehicle, the vehicle control unit 15 outputs a gear change command to the transmission 54.

[0028] The information notification unit 16 generates information to notify the occupants as appropriate depending on the situation, and outputs the generated information to the sound generation device 55 and the display device 56. For example, when the time to collision (TTC) with an obstacle calculated by the driving assistance determination unit 14 falls below a predetermined time (e.g., 3 seconds), the information notification unit 16 outputs a buzzer sound or a display on the meter panel to notify the occupants of the danger.

[0029] The vehicle control device configured as described above makes it possible to predict with high accuracy the behavior of a moving object that interacts with the vehicle. Furthermore, when the vehicle control device controls the traveling of the vehicle, as will be described later with reference to Figures 7 to 12, it is possible to realize driving assistance and automatic driving with a low risk of collision by preventing the target trajectory of the vehicle from interfering with the predicted trajectory of the moving object. Furthermore, when the predicted behavior of the moving object has the potential to impede the traveling of the vehicle, the vehicle control device notifies the driver of the risk, allowing the driver to keep a close eye on the moving object while driving, thereby making it possible to prevent accidents.

[0030] [Hardware Configuration of Computer] Here, the hardware configuration of the computer provided in the vehicle control device 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the computer provided in the vehicle control device 10. A calculator 20 shown in Fig. 2 is an example of hardware used as a computer. The vehicle control device 10 according to this embodiment realizes vehicle control performed by the respective functional blocks shown in Fig. 1 working together as a result of the calculator 20 (computer) executing a program.

[0031] The computer 20 includes a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, and a RAM (Random Access Memory) 23, all connected to a system bus. The computer 20 further includes a non-volatile storage 24 and a communication interface 25.

[0032] The CPU 21 reads out program code of software that realizes each function according to this embodiment from the ROM 22, loads it into the RAM 23, and executes it. Variables, parameters, etc. generated during the calculation processing of the CPU 21 are temporarily written to the RAM 23, and these variables, parameters, etc. are read out as appropriate by the CPU 21. The CPU 21 executes the program code read out from the ROM 22, thereby realizing the functions of each functional block of the vehicle control device 10. However, another processor such as an MPU (Micro Processing Unit) may be used instead of the CPU 21.

[0033] The non-volatile storage 24 is an example of a recording medium, and is capable of storing data used by a program, data obtained by executing a program, etc. The map information storage unit 12 and the model storage unit described above are configured using the non-volatile storage 24. The non-volatile storage 24 may also store an operating system (OS) or a program executed by the CPU 21. The non-volatile storage 24 may be a hard disk drive (HDD), a solid state drive (SSD), an optical or magnetic disk medium, a semiconductor memory card, or the like.

[0034] A communication device such as a network interface card (NIC) is used as the communication interface 25. The communication interface 25 can transmit and receive various data to and from external devices via a communication network such as a CAN connected to a terminal of the NIC or a dedicated line.

[0035] [Mobile Object Behavior Prediction Device] Next, focusing on the mobile object behavior prediction unit 13 shown in FIG. 1, a mobile object behavior prediction device configured specifically for the function thereof will be described with reference to FIGS.

[0036] (First Example of Mobile Object Behavior Prediction Device) Fig. 3 is a schematic configuration diagram showing a first example of a mobile object behavior prediction device in the vehicle control device 10. The mobile object behavior prediction device 200a illustrated in Fig. 3 is a computer that predicts the behavior of a mobile object, and functions as a mobile object behavior prediction unit 13a by executing a program recorded on a recording medium (not shown).

[0037] The mobile object behavior prediction device 200a is connected to a transmission line such as a CAN (not shown) of the host vehicle or a dedicated line. Via these transmission lines, the mobile object behavior prediction device 200a receives time-series information including the positions and speeds of the host vehicle and moving objects around the host vehicle, map information, and host vehicle status information including the status of the host vehicle's lighting devices and indicators and DMS information. The mobile object behavior prediction device 200a then outputs a behavior prediction result of the moving object using the mobile object behavior prediction unit 13a.

[0038] The mobile object behavior prediction unit 13a has the same functions as the mobile object behavior prediction unit 13 (FIG. 1). The mobile object behavior prediction model 131a has the same configuration as the mobile object behavior prediction model 131. The mobile object behavior prediction model 131a will be described in detail below.

[0039] The mobile object behavior prediction model 131a is configured to receive time-series information such as the positions and speeds of the host vehicle and mobile objects around the host vehicle, as well as host vehicle state information including the states of the host vehicle's lighting devices and indicators and DMS information, and to output a behavior prediction result for the mobile object. For example, when machine learning is used, the mobile object behavior prediction model 131a receives time-series information and the host vehicle state information as input, and outputs a behavior prediction result based on a machine learning algorithm. Map information may be included in the input to the mobile object behavior prediction model 131a.

[0040] As described above, the mobile object behavior prediction device (e.g., mobile object behavior prediction device 200a) in the first embodiment is configured to include a model storage unit (e.g., ROM 22 or non-volatile storage 24) and a mobile object behavior prediction unit (e.g., mobile object behavior prediction unit 13a). The model storage unit receives as input time-series information including position information of a first vehicle (e.g., the host vehicle) and moving objects around the first vehicle (e.g., a second vehicle such as an oncoming vehicle, a pedestrian, etc.) at different times, and first vehicle state information (e.g., the host vehicle state information) that is information indicating an operation or state of the first vehicle displayed by the first vehicle or an occupant of the first vehicle and that is recognizable by the moving object, and stores a mobile object behavior prediction model (e.g., mobile object behavior prediction model 131a) that outputs a behavior prediction result of the moving objects around the first vehicle. The mobile object behavior prediction unit inputs time series information including position information of the vehicle and mobile objects around the vehicle obtained at different times, and vehicle state information, which is information representing the operation or state of the vehicle displayed by the vehicle or an occupant of the vehicle and which can be recognized by the mobile object, into a mobile object behavior prediction model, and outputs the behavior prediction result of the mobile object.

[0041] The above configuration makes it possible to predict with high accuracy the behavior of moving objects that interact with the host vehicle. Furthermore, the results of the predicted behavior of moving objects can be used for various purposes, not just for vehicle control as shown in Figure 1.

[0042] (Second Example of Mobile Object Behavior Prediction Device) Next, a second example of the above-mentioned mobile object behavior prediction device will be described with reference to Fig. 4. Fig. 4 is a schematic configuration diagram showing the second example of the mobile object behavior prediction device in the vehicle control device 10. The mobile object behavior prediction device 200b illustrated in Fig. 4 is a computer that predicts the behavior of a mobile object, and functions as the mobile object behavior prediction unit 13b and the host vehicle behavior prediction unit 17 by executing a program recorded on a recording medium (not shown).

[0043] The mobile object behavior prediction device 200b is connected to a transmission line such as a CAN (not shown) of the host vehicle or a dedicated line. Time-series information including the positions and speeds of the host vehicle and moving objects around the host vehicle, map information, and host vehicle status information including the status of the host vehicle's lighting devices and indicators and DMS information are input to the mobile object behavior prediction device 200b via these transmission lines. The mobile object behavior prediction device 200b then outputs a behavior prediction result of the moving object using the mobile object behavior prediction unit 13b.

[0044] The host vehicle behavior prediction unit 17 inputs time-series information such as the position and speed of the host vehicle, map information, and host vehicle state information including the state of the host vehicle's lighting devices and indicators and DMS information, and predicts the future behavior of the host vehicle. For example, the host vehicle behavior prediction unit 17 predicts a sequence of trajectory points that the host vehicle will pass in the future as the future behavior of the host vehicle. Here, similar to the mobile object behavior prediction model 131a in FIG. 3 , the host vehicle behavior prediction unit 17 predicts the behavioral intention of the host vehicle as the future behavior of the host vehicle from the host vehicle state information. Note that the prediction method may be a method using machine learning, a method for searching for an optimal trajectory, or a method using rule-based conditional branching.

[0045] As described above, the second embodiment includes a host vehicle behavior prediction unit 17 that outputs a host vehicle behavior prediction result based on time-series information including host vehicle position information at different times and multi-input host vehicle state information. The host vehicle behavior prediction unit 17 converts the multi-input host vehicle state information into one-dimensional host vehicle behavior information. The moving object behavior prediction unit 13b then inputs the host vehicle behavior prediction result predicted by the host vehicle behavior prediction unit 17 to the moving object behavior prediction model 131b in place of the host vehicle state information, thereby causing the moving object behavior prediction model 131b to output the moving object behavior prediction result.

[0046] The mobile object behavior prediction unit 13b has substantially the same functions as the mobile object behavior prediction unit 13a (FIG. 3). However, instead of inputting host vehicle state information, the mobile object behavior prediction unit 13b inputs the host vehicle behavior prediction result output from the host vehicle behavior prediction unit 17. The mobile object behavior prediction model 131b inputs time-series information such as the position and speed of the host vehicle and moving objects around the host vehicle, as well as the host vehicle behavior prediction result from the host vehicle behavior prediction unit 17, and outputs the mobile object behavior prediction result. Map information may also be input to the mobile object behavior prediction model 131b.

[0047] In general, machine learning has the problem that the computational model becomes complex as the number of inputs increases. In contrast, in the second embodiment, when the mobile object behavior prediction unit 13b uses a mobile object behavior prediction model 131b learned by machine learning, one-dimensional information on the host vehicle behavior can be input to the mobile object behavior prediction model 131b instead of host vehicle state information. This makes it possible to prevent the mobile object behavior prediction model 131b from becoming complex.

[0048] Furthermore, even when the machine-learned mobile object behavior prediction model 131b is not used, the mobile object behavior prediction unit 13b does not need to calculate prediction results for each piece of host vehicle state information using conditional branching, etc. This makes it possible to simplify the algorithm using the mobile object behavior prediction model 131b of the mobile object behavior prediction unit 13b.

[0049] (Third Example of Mobile Object Behavior Prediction Device) Next, a third example of the above-mentioned mobile object behavior prediction device will be described with reference to Fig. 5. Fig. 5 is a schematic configuration diagram showing the third example of the mobile object behavior prediction device in the vehicle control device 10. The mobile object behavior prediction device 200c illustrated in Fig. 5 is a computer that predicts the behavior of a mobile object, and functions as the mobile object behavior prediction unit 13c and the mobile object behavior prediction determination unit 18 by executing a program recorded on a recording medium (not shown).

[0050] The mobile object behavior prediction device 200c is connected to a transmission line such as a CAN (not shown) of the host vehicle or a dedicated line. Via these transmission lines, the mobile object behavior prediction device 200c receives time-series information including the positions and speeds of the host vehicle and moving objects around the host vehicle, map information, and host vehicle status information including the status of the host vehicle's lighting devices and indicators and DMS information. The mobile object behavior prediction device 200c then outputs two behavior prediction results for the moving object using the mobile object behavior prediction unit 13c.

[0051] The mobile object behavior prediction unit 13c has a function similar to that of the mobile object behavior prediction unit 13a ( FIG. 3 ). The mobile object behavior prediction unit 13c includes a model storage unit (not shown), which stores a mobile object behavior prediction model 131c and a mobile object basic behavior prediction model 132 learned using machine learning in the model storage unit. The mobile object behavior prediction unit 13c holds each piece of information input to the mobile object behavior prediction unit 13c as time-series information from the past, and inputs at least this time-series information into the mobile object behavior prediction model 131c and the mobile object basic behavior prediction model 132 to predict the behavior of the mobile object. In this specification, when there is no need to particularly distinguish between the mobile object behavior prediction units 13a to 13c, the reference numerals will be omitted and they will be referred to as "mobile object behavior prediction units."

[0052] Here, the mobile object behavior prediction model 131c has the same function as the mobile object behavior prediction model 131a ( FIG. 3 ). The mobile object basic behavior prediction model 132 differs from the mobile object behavior prediction model 131c in that it does not input host vehicle state information. In other words, the mobile object basic behavior prediction model 132 is a conventional prediction model that uses time-series information of the host vehicle and the mobile object to output a behavior prediction result of the mobile object.

[0053] Here, it is desirable to output reliability information in addition to the behavior prediction results from the mobile object behavior prediction model 131c and the mobile object basic behavior prediction model 132 as information for determining the behavior prediction of the mobile object in the subsequent mobile object behavior prediction determination unit 18. Furthermore, the behavior prediction results may be output from the mobile object behavior prediction model 131c only when, among the host vehicle state information, information that determines the behavioral intention of the host vehicle, such as that described in the host vehicle behavior prediction unit 17 (FIG. 4), is input.

[0054] The mobile object behavior prediction determination unit 18 finally determines the behavior prediction of the mobile object based on the respective behavior prediction results output from the mobile object behavior prediction model 131c and the mobile object basic behavior prediction model 132. As a method for determining the behavior prediction of the mobile object, for example, there is a method of comparing information on the reliability of the behavior prediction results output from both prediction models and adopting the behavior prediction result with the higher reliability.

[0055] As described above, in the third embodiment, the mobile object behavior prediction unit 13c causes the mobile object basic behavior prediction model 132 to output a mobile object behavior prediction result (an example of a first prediction result), and causes the mobile object behavior prediction model 131c to output a mobile object behavior prediction result (an example of a second prediction result). Then, the mobile object behavior prediction determination unit 18 determines the behavior prediction of the mobile object based on the first prediction result and the second prediction result.

[0056] In the third embodiment, the first prediction result includes first reliability information about the first prediction result, and the second prediction result includes second reliability information about the second prediction result. The mobile object behavior prediction determination unit 18 determines the behavior prediction of the mobile object based on the first reliability information and the second reliability information. Note that, as another method for determining the behavior prediction of the mobile object, when a behavior prediction result is output from only one of the prediction models, the output behavior prediction result may be adopted. The method for determining the behavior prediction of the mobile object from multiple behavior prediction results is not limited to the above example.

[0057] Incidentally, when using a machine-learned predictive model to predict the behavior of a moving object, if the training data is biased, such as when the amount of data is small, problems such as overfitting can occur. Overfitting occurs when the accuracy rate is high on the training data, but when it is used on actual data (in the case of a car, data obtained while driving), the accuracy rate drops, and the model is optimized only for the training data.

[0058] Here, in order to predict the behavior of a moving object, it is desirable to acquire a large amount of learning data targeting a variety of scenes. However, it is expected that there will be very few scenes in which interactions between the host vehicle and a moving object occur. For this reason, as shown in FIG. 5 , by dividing the prediction model and configuring one prediction model without inputting host vehicle state information, overlearning can be prevented. Furthermore, since the moving object behavior prediction model 131c is a prediction model that is specifically trained when host vehicle state information is input, it is possible to output stable prediction results.

[0059] It is also possible to incorporate the host vehicle behavior prediction unit 17 shown in Fig. 4 into the configuration of the moving object behavior prediction device 200c in Fig. 5. In this case, the host vehicle behavior prediction unit 17 is disposed between the input terminal of the moving object behavior prediction device 200c for host vehicle state information and the moving object behavior prediction model 131. By using such a configuration, when using the moving object behavior prediction model 131c learned by machine learning, it is possible to avoid the moving object behavior prediction model 131c becoming complicated.

[0060] Furthermore, even when the machine-learned mobile object behavior prediction model 131c is not used, the mobile object behavior prediction unit 13c does not need to calculate prediction results for each piece of host vehicle state information using conditional branching, etc. This makes it possible to simplify the algorithm using the mobile object behavior prediction model 131c of the mobile object behavior prediction unit 13c.

[0061] As can be seen from the third embodiment of the mobile object behavior prediction device described above, the behavior prediction result of a mobile object output from the mobile object behavior prediction model differs depending on whether or not host vehicle state information is input to the mobile object behavior prediction model. In the third embodiment, host vehicle state information is not input to the mobile object basic behavior prediction model 132, and host vehicle state information is input only to the mobile object behavior prediction model 131c. In addition, in the first to third embodiments, the behavior prediction result of a mobile object also differs depending on the type of host vehicle state information input to the mobile object behavior prediction model (such as the state of the host vehicle's lighting devices, the state of the host vehicle's display devices, the behavior of the vehicle's occupants, or a combination of these pieces of information). Then, when the driving assistance determination unit 14 uses these mobile object behavior prediction results in determining driving assistance, the content of driving assistance also changes.

[0062] [Overall Processing of Vehicle Control Device] Next, the overall processing of the vehicle control device 10 according to this embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the procedure of the overall processing of the vehicle control device 10.

[0063] First, in processing step S1, the vehicle control device 10 executes an external environment recognition result acquisition process. In this process, the vehicle control device 10 acquires the external environment recognition result (time-series information for the past T seconds) from the external environment recognition device 1 installed in the host vehicle.

[0064] Next, in processing step S2, the vehicle control device 10 executes a vehicle information / map information acquisition process. In this process, the vehicle control device 10 acquires vehicle information such as the vehicle speed, steering angle, yaw rate, lighting device status, display device status, and DMS information of the host vehicle, as well as map information stored in the map information storage unit 12.

[0065] Next, in processing step S3, the vehicle control device 10 executes surrounding environment recognition processing. In this processing, the surrounding environment recognition unit 11 of the vehicle control device 10 detects the type and position of obstacles around the vehicle, the moving direction and moving speed of moving objects among the obstacles, etc. (surrounding environment information of the vehicle) based on the external environment recognition result acquired in processing step S1.

[0066] Next, in processing step S4, the vehicle control device 10 executes a moving object behavior prediction process. In this process, each of the moving object behavior prediction devices 200a to 200c (FIGS. 3 to 5) of the vehicle control device 10 executes a moving object behavior prediction (trajectory information up to n seconds ahead) based on the vehicle information and map information acquired in processing step S2 and the host vehicle surrounding environment information detected in processing step S3. In other words, the moving object behavior prediction devices 200a to 200c execute a moving object behavior prediction based on time-series information, map information, and host vehicle state information about the host vehicle and / or moving objects around the host vehicle.

[0067] Next, in processing step S5, the vehicle control device 10 executes a driving assistance determination process. In this process, the driving assistance determination unit 14 of the vehicle control device 10 determines appropriate driving assistance based on the vehicle information and map information acquired in processing step S2, the vehicle surrounding environment information detected in processing step S3, and the behavior prediction result of the moving object calculated in processing step S4.

[0068] Next, in processing step S6, the vehicle control device 10 executes a vehicle control process. In this process, the vehicle control unit 15 of the vehicle control device 10 controls the running of the host vehicle based on the content of the driving assistance determined in processing step S5.

[0069] Next, in processing step S7, the vehicle control device 10 executes information notification processing. In this processing, the information notification unit 16 of the vehicle control device 10 outputs information to be notified to the occupants of the vehicle based on the content of the driving assistance determined in processing step S5, and the series of processing ends. The vehicle control device 10 periodically repeats the series of processing described above.

[0070] [Input / Output Information of Mobile Object Behavior Prediction Unit] Next, the input information and output information of the mobile object behavior prediction unit 13 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the input information and output information in the mobile object behavior prediction process. Fig. 7 shows an example of a situation in which the host vehicle 70 and a vehicle 71, which is a mobile object, are approaching a crossroads intersection. The vehicle 71 is about to enter the intersection from the left as viewed from the host vehicle 70.

[0071] Trajectory point sequences 75 and 76 are past travel trajectories of the host vehicle 70 and vehicle 71, respectively. When a prediction model using machine learning is used to predict mobile object behavior, information on these past travel trajectories is stored in chronological order and used as input information. Trajectory point sequences 72, 73, and 74 are future travel trajectory point sequences predicted by the mobile object behavior prediction unit and used as output information. In this example, three trajectory point sequences are predicted, and reliability information can be added to each of them before output.

[0072] In this embodiment, the behavior prediction result of a moving object output from the moving object behavior prediction unit 13 is predicted trajectory information (e.g., a sequence of trajectory points) including the future position and / or future speed of the moving object. The behavior prediction result of a moving object also includes reliability information about the predicted trajectory information, and includes at least one or more pieces of predicted trajectory information for one moving object.

[0073] According to this embodiment, by using the behavior prediction results of the moving body, which include one or more predicted trajectory information, in the downstream driving assistance judgment unit 14, more appropriate driving assistance can be achieved based on behavior prediction results that are more accurate than conventional ones.

[0074] [Specific Operation of the Mobile Object Behavior Prediction Unit] Next, an example of a specific operation of the mobile object behavior prediction unit will be described with reference to FIGS. 8 to 12. FIG.

[0075] (First Example of Mobile Object Behavior Prediction Processing) FIG. 8 is a diagram showing a first example of the mobile object behavior prediction processing. FIG. 8 shows an example of a situation in which the host vehicle 80 and a mobile vehicle 81 are approaching an intersection 89 in a parking lot. In the upper part of FIG. 8, the vehicle 81 has entered the parking lot and is stopped at the entrance to the intersection 89, and the host vehicle 80 is traveling toward the intersection 89 (state (1)). In this situation, when the mobile object behavior prediction unit predicts the future behavior of the vehicle 81, two trajectory point sequences 82 and 83 can be predicted as candidates. Furthermore, the vehicle 81 has its right turn signal 81d on. If the host vehicle 80 can detect this state, the reliability of the trajectory point sequence 82 can be set high and output.

[0076] However, in this state, when the vehicle 81 looks at the host vehicle 80, it is not possible to grasp the behavioral intention of the host vehicle 80 (whether the host vehicle 80 will go straight through the intersection 89, turn left, or take some other action). For this reason, it is not possible to increase the reliability of the speed information of the trajectory point sequence 82 (the timing at which the host vehicle 80 will enter the intersection 89), and ultimately the trajectory point sequence 82 remains merely a prediction with low reliability. For example, if the vehicle 81 performs self-driving within a parking lot as driving assistance in this state, it is not possible to gauge the timing at which the vehicle 81 will enter the intersection 89, and therefore the host vehicle 80 will decelerate early and approach the intersection 89 slowly for safety reasons.

[0077] The lower part of Fig. 8 shows the situation after a predetermined time has elapsed from the upper part of Fig. 8, in which the left turn signal 80d of the host vehicle 80 is on (state (2)). At this time, when the host vehicle 80 is viewed from the vehicle 81, it can be recognized that the left turn signal 80d of the host vehicle 80 is on, and therefore it can be seen that the state of the vehicle 81 is a predictable state in which "the host vehicle 80 will turn left at the intersection 89 as shown by the dashed line 84."

[0078] In this state, the prediction result for the vehicle 81 by the moving object behavior prediction unit of the host vehicle 80 is the most reliable for the trajectory point sequence 82 by using information from the turn signal 80d of the host vehicle 80, and furthermore, the speed information of the trajectory point sequence 82 can also be predicted with high reliability. Therefore, the moving object behavior prediction unit of the host vehicle 80 can perform prediction of the vehicle 81 with high accuracy. Note that, as a specific example of the speed information of the trajectory point sequence 82, when the distance between the host vehicle 80 and the vehicle 81 is large as shown in the lower part of FIG. 8 , it is possible to predict that the vehicle 81 will enter the intersection 89 before the host vehicle 80 arrives at the intersection 89.

[0079] As described above, if the vehicle control device 10 (mobile body behavior prediction unit) of the vehicle 80 can predict the behavior of the vehicle 81 with high accuracy, then, for example, as a driving assistance, vehicles interacting with each other during automated driving in parking lots, narrow roads, etc. can drive smoothly with each other.

[0080] (Second Example of Mobile Object Behavior Prediction Processing) Fig. 9 is a diagram showing a second example of the mobile object behavior prediction processing. Fig. 9 shows an example of a situation in which a host vehicle 90 is parked in a parking lot with its hazard lights 90h on, attempting to park in a parking space 95, and an oncoming vehicle 91 is traveling at a constant speed toward the host vehicle 90 through a passage that has become narrow due to construction work.

[0081] In this situation, when the mobile object behavior prediction unit of the host vehicle 90 predicts the future behavior of the oncoming vehicle 91, three trajectory point sequences are predicted: a trajectory point sequence 92 in which the host vehicle 90 stops temporarily just before a construction sign 96, a trajectory point sequence 93 in which the host vehicle 90 goes straight ahead avoiding the construction sign 96, and a trajectory point sequence 94 in which the host vehicle 90 turns right at an intersection 99 and leaves the parking lot. Here, by using information on the illumination status of the hazard lights 90h of the host vehicle 90, which can be recognized by the oncoming vehicle 91, it is determined that the state of the oncoming vehicle 91 is a predictable state in which "the host vehicle 90 wants to park but is stopped so as not to obstruct the passage of the oncoming vehicle 91."

[0082] Therefore, among the trajectory point sequences 92 to 94 predicted by the moving object behavior prediction unit, the reliability of trajectory point sequence 93, in which the oncoming vehicle 91 travels straight while avoiding the under-construction signboard 96, is the highest. The speed information of trajectory point sequence 93 makes it possible to predict that the oncoming vehicle 91 will travel while maintaining approximately the current constant speed (while appropriately reducing speed where the passage width narrows).

[0083] Note that the reliability of the trajectory point sequence 92, which stops just before the under construction sign 96, is set low because the distance to the under construction sign 96 is short and rapid deceleration is required to stop. Similarly, the reliability of the trajectory point sequence 94, which turns right at an intersection 99, is set low because the illumination of the right turn signal of the oncoming vehicle 91 is not detected and the speed is maintained.

[0084] As described above, if the vehicle control device 10 (mobile object behavior prediction unit) of the host vehicle 90 can predict the behavior of the oncoming vehicle 91 with high accuracy, for example, when performing automatic parking as driving assistance, the timing to start control can be changed based on the degree of proximity of the oncoming vehicle 91. Therefore, the host vehicle 90 can perform smooth parking control without interfering with the driving of the oncoming vehicle 91.

[0085] (Third Example of Mobile Object Behavior Prediction Processing) Fig. 10 is a diagram showing a third example of the mobile object behavior prediction processing. Fig. 10 shows an example in which, in a parking lot, the host vehicle 100 is about to go straight through an intersection 109, and an oncoming vehicle 101 is about to turn right at the intersection 109. In this situation, the host vehicle 100 has confirmed the presence of the oncoming vehicle 101 entering the parking space, and is about to decelerate (dashed line 103) and stop just before the intersection 109 while flashing its headlights 100f to give way to the oncoming vehicle 101.

[0086] In this situation, when the mobile object behavior prediction unit predicts the future behavior of the oncoming vehicle 101, two trajectory point sequences are predicted: a trajectory point sequence 104 for going straight through the intersection 109, and a trajectory point sequence 105 for turning right at the intersection 109. At this time, if the host vehicle 100 has already detected that the right turn indicator 101d of the oncoming vehicle 101 is illuminated, the reliability of the trajectory point sequence 105 for turning right at the intersection 109 is set to be high.

[0087] Here, by using the flashing state of the headlights 100f of the host vehicle 100, which can be recognized by the oncoming vehicle 101, it is possible to predict that the state of the oncoming vehicle 101 is "the host vehicle 100 is trying to give way." Therefore, among the trajectory point sequences 94 to 95 predicted by the moving object behavior prediction unit, the trajectory point sequence 105 for turning right at the intersection 109 has the highest reliability. The speed information of the trajectory point sequence 105 makes it possible to predict that the oncoming vehicle 101 will enter the intersection 109 around the time the host vehicle 100 stops.

[0088] As described above, if the vehicle control device 10 (mobile body behavior prediction unit) of the vehicle 100 can predict the behavior of the oncoming vehicle 101 with high accuracy, then, for example, as a driving assistance, vehicles interacting with each other during automated driving in parking lots, narrow roads, etc. can drive smoothly with each other.

[0089] (Fourth Example of Mobile Object Behavior Prediction Processing) Fig. 11 is a diagram showing a fourth example of the mobile object behavior prediction processing. Fig. 11 shows a situation similar to Fig. 10 , in which the host vehicle 110 is about to go straight through an intersection 119 in a parking lot, and the oncoming vehicle 101 is about to turn right at the intersection 119. Here, instead of signaling with flashing lights with the headlights 100f, the driver of the host vehicle 110 is making a gesture (e.g., a signal with his hand) meaning "you may go ahead" to the driver of the oncoming vehicle 111, and is about to slow down (dashed line 113) and stop just before the intersection 119.

[0090] In this situation, when the mobile object behavior prediction unit of the vehicle 110 predicts the future behavior of the oncoming vehicle 111, two trajectory point sequences are predicted: a trajectory point sequence 114 for going straight through the intersection 119, and a trajectory point sequence 115 for turning right at the intersection. At this time, if the vehicle 110 has already detected that the right turn indicator 111d of the oncoming vehicle 111 is illuminated, the reliability of the trajectory point sequence 115 for turning right at the intersection 119 is set to be high.

[0091] Here, by using the state of the DMS information detecting a gesture of the driver of the host vehicle 110 that can be recognized by the oncoming vehicle 111, it is possible to predict that the state of the oncoming vehicle 111 is "the host vehicle 110 is trying to give way." Therefore, among the trajectory point sequences 114 to 115 predicted by the mobile object behavior prediction unit, the trajectory point sequence 115 for turning right at the intersection 119 has the highest reliability. The speed information of the trajectory point sequence 115 makes it possible to predict that the oncoming vehicle 111 will enter the intersection 119 around the time the host vehicle 110 stops.

[0092] As described above, if the vehicle control device 10 (mobile body behavior prediction unit) of the vehicle 100 can predict the behavior of the oncoming vehicle 111 with high accuracy, then, for example, as a driving assistance, vehicles interacting with each other during automated driving in parking lots, narrow roads, etc. can drive smoothly with each other.

[0093] (Fifth Example of Mobile Object Behavior Prediction Processing) Fig. 12 is a diagram showing a fifth example of the mobile object behavior prediction processing. Fig. 12 shows an example of a situation in which, on a road with one lane in each direction, a vehicle 120 is traveling at a low speed on the shoulder of the road with a message "Slow speed work in progress. Please drive with caution" displayed on a display in the rear window (dashed line 123), and a following vehicle (hereinafter referred to as "following vehicle") 121 is approaching.

[0094] In this situation, when the mobile object behavior prediction unit predicts the future behavior of the following vehicle 121, two trajectory point sequences are predicted: a trajectory point sequence 124 in which the following vehicle 121 travels while maintaining a safe distance from the host vehicle 120, and a trajectory point sequence 125 in which the following vehicle 121 overtakes the host vehicle 120.

[0095] Here, by using the display state of the display 122 of the own vehicle 120, which can be recognized by the following vehicle 121, it is possible to predict that the state of the following vehicle 121 is a state in which "the own vehicle 120 will continue to travel on the shoulder of the road at a low speed." Therefore, among the trajectory point sequences 124 to 125 predicted by the mobile object behavior prediction unit, the trajectory point sequence 125 for overtaking the own vehicle 120 has the highest reliability. Regarding the speed information of the trajectory point sequence 125, for example, if there is no oncoming vehicle approaching, it is possible to predict that the following vehicle 121 will travel at a speed close to the speed limit of the road. Furthermore, if an oncoming vehicle is present, it is possible to predict that the following vehicle 121 will travel at a speed higher than the speed limit based on the positional relationship with the oncoming vehicle.

[0096] As described above, if the vehicle control device 10 (mobile body behavior prediction unit) of the vehicle 120 can predict the behavior of the following vehicle 121 with high accuracy, it will be possible to achieve smooth driving that does not interfere with surrounding traffic flow in automated driving, for example, as a driving assistance function.

[0097] As explained above using multiple examples and multiple examples of moving body behavior prediction processing, according to one embodiment of the present invention, by taking into account vehicle state information when predicting the behavior of a moving body, it is possible to predict with high accuracy the behavior of a moving body that interacts with the vehicle.

[0098] The above-described host vehicle status information may have different meanings depending on the region, country, etc. For example, flashing headlights may be used to signal to the other vehicle to give way, or to signal that the driver wants to go ahead. Hazard lights may be used to signal when parking or stopping, or to signal an emergency. The usage of such host vehicle status information may differ depending on the region, country, etc. Therefore, when using a mobile object behavior prediction model using machine learning, the mobile object behavior prediction model is trained based on host vehicle status information defined for each region of the target country. It is desirable that the mobile object behavior prediction unit determines the region in which the host vehicle is located from map information, etc., and further inputs information indicating the region into the prediction model to predict the behavior of the mobile object. The same applies to the prediction model used by the host vehicle behavior prediction unit 17.

[0099] Alternatively, it is desirable to prepare multiple mobile object behavior prediction models trained for each region of the target country and predict the behavior of the mobile object while switching to a mobile object behavior prediction model appropriate for the region. The mobile object behavior prediction unit determines information indicating the region in which the host vehicle is located from map information, etc., and further inputs information indicating the region in which the host vehicle is located. Then, the mobile object behavior prediction unit selects a mobile object behavior prediction model appropriate for the region from the multiple mobile object behavior prediction models based on the information indicating the region, and outputs a behavior prediction result of the mobile object using the selected mobile object behavior prediction model. In this way, the mobile object behavior prediction unit predicts the behavior of the mobile object while switching to a mobile object behavior prediction model appropriate for the region in which the host vehicle is located. The same applies to the prediction model used by the host vehicle behavior prediction unit 17.

[0100] Furthermore, when a prediction model using machine learning is used in the moving object behavior prediction unit, the description has been given assuming that the trained prediction model is implemented in a vehicle control device or the like, but this example is not limited thereto. A conceivable method is to upload data from each vehicle's travelling state to a cloud or the like, extract data necessary for learning from that data, re-learn, and download the prediction model to the vehicle for execution. For example, a conceivable method is to determine data on the cloud when an interaction occurs between the moving object to which the host vehicle status information is input, add that data, perform re-learning, and download the data to the vehicle for execution. This can improve the accuracy of predicting the behavior of the moving object when there is an interaction with the host vehicle.

[0101] Although the present embodiment has been described using several patterns as examples, the present invention can also be applied to other patterns. In particular, the present embodiment has been described focusing on driving scenes in parking lots, but the present invention can also be applied to places such as ordinary roads, expressways, other dedicated roads, and private land.

[0102] As described above, the present invention is not limited to the above-described embodiments, and various other modifications and applications are possible without departing from the spirit of the invention as set forth in the claims. For example, the above-described embodiments have been described in detail and specifically to clearly explain the present invention, and are not necessarily limited to those including all of the components described. Furthermore, it is also possible to add, replace, or delete other components to or from part of the configuration of the embodiments.

[0103] Furthermore, some or all of the above-described configurations, functions, processing units, etc. may be implemented in hardware, for example, by designing them as integrated circuits, etc. As the hardware, a broad processor device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used.

[0104] In the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other.

[0105] REFERENCE SIGNS LIST 1...External environment recognition device, 10...Vehicle control device, 11...Surrounding environment recognition unit, 12...Map information storage unit, 13...Mobile object behavior prediction unit, 13a to 13c...Mobile object behavior prediction unit, 14...Driving assistance determination unit, 15...Vehicle control unit, 16...Information notification unit, 17...Own vehicle behavior prediction unit, 18...Mobile object behavior prediction determination unit, 131...Mobile object behavior prediction model, 131a to 131c...Mobile object behavior prediction model, 132...Mobile object basic behavior prediction model, 200a to 200c...Mobile object behavior prediction device

Claims

1. A mobile object behavior prediction device comprising: a model storage unit that stores a mobile object behavior prediction model that receives as input time series information including position information of the host vehicle and mobile objects around the host vehicle at different times, and host vehicle state information that is information that represents the operation or state of the host vehicle and is displayed by the host vehicle or a passenger of the host vehicle and is information that can be recognized by the mobile object, and outputs a behavior prediction result for the mobile object around the host vehicle; and a mobile object behavior prediction unit that receives as input time series information including position information of the host vehicle and mobile objects around the host vehicle obtained at different times, and the host vehicle state information that is information that represents the operation or state of the host vehicle and is displayed by the host vehicle or a passenger of the host vehicle and is information that can be recognized by the mobile object, into the mobile object behavior prediction model, and outputs a behavior prediction result for the mobile object.

2. A mobile body behavior prediction device as described in claim 1, comprising a vehicle behavior prediction unit that outputs a behavior prediction result of the vehicle based on time series information including position information of the vehicle at different times and the vehicle state information, wherein the mobile body behavior prediction unit inputs the behavior prediction result of the vehicle predicted by the vehicle behavior prediction unit into the mobile body behavior prediction model in place of the vehicle state information, thereby causing the mobile body behavior prediction model to output the behavior prediction result of the moving body.

3. A mobile body behavior prediction device as described in claim 1, further comprising a mobile body behavior prediction determination unit that determines a behavior prediction of the mobile body, wherein the model storage unit further stores a mobile body basic behavior prediction model that outputs a behavior prediction result of the mobile body based on the time series information, the mobile body behavior prediction unit causes the mobile body basic behavior prediction model to output a first prediction result of the mobile body and causes the mobile body behavior prediction model to output a second prediction result of the mobile body, and the mobile body behavior prediction determination unit determines a behavior prediction of the mobile body based on the first prediction result and the second prediction result.

4. The mobile object behavior prediction device according to claim 1, wherein the vehicle state information includes at least one of the state of the vehicle's lighting devices, the state of the vehicle's display devices, or the behavior of the vehicle's occupants.

5. A mobile object behavior prediction device according to claim 1, wherein the behavior prediction result of the mobile object output from the mobile object behavior prediction model differs depending on the presence or absence or type of the host vehicle state information input to the mobile object behavior prediction model.

6. A mobile body behavior prediction device as described in claim 1, wherein the behavior prediction result is predicted trajectory information including the future position and / or future speed of the mobile body, includes reliability information about the predicted trajectory information, and includes at least one predicted trajectory information for one mobile body.

7. The mobile object behavior prediction device according to claim 1, wherein the mobile object behavior prediction model receives the time series information and the host vehicle state information as inputs, and outputs the behavior prediction result based on a machine learning algorithm.

8. A mobile object behavior prediction device as described in claim 3, wherein the first prediction result includes first reliability information about the first prediction result, the second prediction result includes second reliability information about the second prediction result, and the mobile object behavior prediction determination unit determines the behavior prediction of the mobile object based on the first reliability information and the second reliability information.

9. A mobile object behavior prediction device as described in claim 7, wherein the mobile object behavior prediction model is trained based on the vehicle state information defined for each specified region, and further inputs information indicating the region in which the vehicle is located, and outputs the behavior prediction result of the mobile object.

10. The mobile object behavior prediction device according to claim 7, wherein the model storage unit includes a plurality of the mobile object behavior prediction models trained for each predetermined region, and the mobile object behavior prediction unit further inputs information indicating the region in which the vehicle is located, selects a mobile object behavior prediction model from the plurality of mobile object behavior prediction models based on the information indicating the region, and outputs a behavior prediction result for the mobile object using the selected mobile object behavior prediction model.

11. A vehicle control device comprising the mobile object behavior prediction device according to claim 1, which controls the running of the vehicle using the results of the predicted behavior of the mobile object output from the mobile object behavior prediction device.

12. A mobile object behavior prediction method using a mobile object behavior prediction device comprising a model storage unit and a mobile object behavior prediction unit, wherein the model storage unit stores a mobile object behavior prediction model that receives as input time series information including position information of the host vehicle and mobile objects around the host vehicle at different times and host vehicle state information that is information that represents the operation or state of the host vehicle and is displayed by the host vehicle or an occupant of the host vehicle and is information that is recognizable by the moving object, and outputs a behavior prediction result of the moving object around the host vehicle, and the mobile object behavior prediction unit inputs the time series information including position information of the host vehicle and mobile objects around the host vehicle obtained at different times and the host vehicle state information that is information that represents the operation or state of the host vehicle and is displayed by the host vehicle or an occupant of the host vehicle and is information that is recognizable by the moving object to the mobile object behavior prediction model, and outputs a behavior prediction result of the moving object.

Images