Prediction device, prediction method, program, and vehicle control system

Abstract

To further accurately predict a future location of a mobile.SOLUTION: A prediction device comprises a mobile information acquisition unit, a cumulative map generation unit and a prediction unit. The mobile information acquisition unit acquires mobile information indicating locations of one or more mobiles including a first mobile defined as a prediction object. The cumulative map generation unit generates cumulative map information representing a plurality of locations indicated by the mobile information acquired at a plurality of first times prior to a reference time on a map. The prediction unit predicts the location of the first mobile at a second time after the reference time from environment map information representing an environment around the first mobile at the reference time on a map, mobile information acquired at the reference time, and the cumulative map information.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] An embodiment of the present invention relates to a prediction device, a prediction method, a program, and a vehicle control system. [Background technology]

[0002] To realize safe and comfortable automatic driving and driving assistance for automobiles, as well as autonomous movement of robots, it is necessary to grasp in advance not only static obstacles such as buildings, fences, and curbs, but also the movements of moving objects such as other vehicles and pedestrians. In particular, in scenes such as lane changing and crossing intersections, it is necessary to predict how moving objects will move in the future in order to determine the safe behavior of the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-186446 A [Non-patent literature]

[0004] [Non-Patent Document 1] Cui, Henggang, et al. "Multimodal trajectory predictions for autonomous driving using deep convolutional networks." International Conference on Robotics and Automation (ICRA). IEEE, 2019. Summary of the Invention [Problem to be solved by the invention]

[0005] There is a demand for predictive technology that can more accurately predict the future positions of moving objects. [Means for solving the problem]

[0006] The prediction device of the embodiment includes a mobile object information acquisition unit, a cumulative map generation unit, and a prediction unit. The mobile object information acquisition unit acquires mobile object information indicating the positions of one or more mobile objects including a first mobile object to be predicted. The cumulative map generation unit generates cumulative map information that indicates, on a map, a plurality of positions indicated by the mobile object information acquired at a plurality of first times before a reference time. The prediction unit predicts the position of the first mobile object at a second time after the reference time from environmental map information that indicates, on a map, the environment around the first mobile object at the reference time, the mobile object information acquired at the reference time, and the cumulative map information. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing a vehicle according to an embodiment. [Diagram 2] FIG. 2 is a hardware configuration diagram of a processing device. [Diagram 3] FIG. 13 is a diagram showing an example of predicting a trajectory according to map information prepared in advance. [Figure 4] FIG. 13 is a diagram showing an example of predicting a trajectory using only sensor information. [Diagram 5] FIG. 2 is a functional block diagram of a prediction device. [Figure 6] FIG. 4 is a diagram for explaining an outline of a trajectory prediction process. [Figure 7] FIG. 4 is a functional block diagram of an environment map generating unit. [Figure 8] FIG. 4 is a diagram showing an example of an obstacle map. [Figure 9] FIG. 11 is a diagram showing an example of a generated accumulation map. [Figure 10] FIG. 4 is a functional block diagram of a prediction unit. [Figure 11] FIG. 11 is a diagram showing an example of normalization of map information. [Figure 12] FIG. 4 is a functional block diagram of a prediction unit. [Figure 13] 13 is a flowchart of a prediction process. [Figure 14] 13 is a flowchart of a learning process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a prediction device according to the present invention will be described in detail below with reference to the accompanying drawings.

[0009] Prediction technology that predicts the future positions of nearby moving objects has been studied in the past. In order to perform more stable predictions, a method has been proposed that uses high-precision maps and aerial photographs that include information such as lane centerlines as environmental information. However, such map information needs to be prepared in advance, and it is not realistic to prepare it for all roads. In addition, generating high-precision maps requires a large processing load and costs.

[0010] Therefore, there is a demand for predicting the positions of other moving bodies in the vicinity using only information obtained from sensors such as cameras and laser sensors attached to moving bodies (automobiles, autonomous mobile robots, etc.) without using map information prepared in advance.

[0011] In addition, in order to perform robust predictions in a wide variety of scenes, a technology has been proposed that collects data for each scene and performs learning. However, this requires a server and storage for data collection. For this reason, there is a demand for a technology that can be used for general purposes with a single learning process.

[0012] The prediction device according to this embodiment generates an environment map (environmental map information) and an accumulation map (accumulation map information) as map information from only information obtained from a sensor, without preparing map information in advance, and predicts the future position of a moving object using the generated map information. The accumulation map is map information that indicates an area through which multiple moving objects, which may include moving objects other than the moving object whose position is to be predicted, are likely to pass. This allows robust predictions to be made even in places where there is no map information or in places visited for the first time.

[0013] The moving body is, for example, a vehicle such as an automobile or a motorbike that moves along a lane provided on a road. The moving body is not limited to an automobile or a motorbike, and may be, for example, a robot that moves along a lane. The moving body may also be an object that moves along a lane on water, such as a ship. In the following, a case where the moving body is a vehicle will be mainly described as an example.

[0014] FIG. 1 is a diagram showing a vehicle 10 according to an embodiment. The vehicle 10 is equipped with a processing device 12. The processing device 12 is, for example, a device including a dedicated or general-purpose computer. At least some functions of the processing device 12 may not be installed in the vehicle 10, but may be installed in another device such as a cloud connected to the vehicle 10 via a network. The vehicle 10 may be, for example, a normal vehicle that runs in response to a driving operation by a person, or may be an autonomous vehicle that can run automatically (autonomously) without being driven by a person. The processing device 12 may be provided not only in the vehicle 10, but also in another device such as a roadside unit.

[0015] 2 is a diagram illustrating an example of a hardware configuration of the processing device 12 according to the embodiment. The processing device 12 includes a storage device 21, an input device 22, a display device 23, a sensor device 24, a communication device 25, a vehicle control device 26, and an information processing device 30.

[0016] The storage device 21 is a hard disk drive, an optical disk drive, or a semiconductor memory element such as a flash memory, etc. The storage device 21 stores programs executed by the processing device 12 and data used by the processing device 12.

[0017] The input device 22 receives instructions and information input from a user and is, for example, an operation panel, a pointing device such as a mouse or a trackball, or an input device such as a keyboard.

[0018] The display device 23 displays various information to the user. The display device 23 is, for example, a display device such as a liquid crystal display device.

[0019] The sensor device 24 has one or more sensors that detect the surrounding conditions of the vehicle 10. For example, the sensor device 24 detects the position, speed, acceleration, angular velocity, and angular acceleration of a moving object (e.g., another vehicle) present around the vehicle 10. Furthermore, the sensor device 24 detects direction indication information that indicates the traveling direction of the moving object present around the vehicle 10. For example, the sensor device 24 has a distance sensor (laser sensor, LiDAR, etc.) that detects distance using laser light. The sensor device 24 may have a millimeter wave sensor that detects the position and speed of the moving object. In addition, the sensor device 24 may have a sonar that detects the distance to a surrounding object by sound waves. In addition, for example, the sensor device 24 may have a camera that captures an image of a surrounding object. The camera may be any type of camera, such as a monocular camera or a stereo camera.

[0020] The communication device 25 transmits and receives information to and from an external device by wireless communication. The communication device 25 acquires detection results of the position, speed, acceleration, angular velocity, angular acceleration, and direction indication information of a moving object existing around the vehicle 10 by a sensor provided in an external device (e.g., a roadside unit) of the vehicle 10. The communication device 25 may also acquire the position, speed, acceleration, angular velocity, angular acceleration, and direction indication information of a moving object by directly communicating with the moving object existing around the vehicle 10, for example, by performing vehicle-to-vehicle communication.

[0021] The vehicle control device 26 controls a drive mechanism for driving the vehicle 10. For example, when the vehicle 10 is an autonomous vehicle, the vehicle control device 26 judges the surrounding situation based on the position of the moving body predicted by the prediction device 40, information obtained from the sensor device 24, and other information, and controls the amount of acceleration, the amount of braking, the steering angle, etc. When the vehicle 10 is a normal vehicle that is driven by a human, the vehicle control device 26 controls the amount of acceleration, the amount of braking, the steering angle, etc. in accordance with the operation information.

[0022] The information processing device 30 is, for example, one or more dedicated or general-purpose computers. The information processing device 30 manages and controls the storage device 21, the input device 22, the display device 23, the sensor device 24, the communication device 25, and the vehicle control device 26. The information processing device 30 has a memory 31 and one or more hardware processors 32.

[0023] The memory 31 includes, for example, a read only memory (ROM) 33 and a random access memory (RAM) 34. The ROM 33 stores programs used to control the information processing device 30 and various setting information in a non-rewritable manner. The RAM 34 is a volatile storage medium such as a synchronous dynamic random access memory (SDRAM). The RAM 34 functions as a working area for one or more hardware processors 32.

[0024] The one or more hardware processors 32 are connected to the memory 31 (ROM 33 and RAM 34) via a bus. The one or more hardware processors 32 may include, for example, one or more central processing units (CPUs) or one or more graphics processing units (GPUs). The one or more hardware processors 32 may also include a semiconductor device or the like including a dedicated processing circuit for implementing a neural network.

[0025] The one or more hardware processors 32 execute various processes in cooperation with various programs stored in advance in the ROM 33 or the storage device 21, using a predetermined area of ​​the RAM 34 as a working area, thereby functioning as a prediction device 40. It can also be interpreted that the information processing device 30 or the processing device 12 corresponds to the prediction device 40. The function of the prediction device 40 will be described in detail later.

[0026] The program for functioning as the prediction device 40 may be configured to be provided as a computer program product by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0027] Furthermore, the program may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the program may be configured to be provided or distributed via a network such as the Internet.

[0028] Here, an example of map information will be described. Fig. 3 is a diagram showing an example of predicting a trajectory of a vehicle (an example of a moving body) according to map information prepared in advance including lane information. Fig. 4 is a diagram showing an example of predicting a trajectory using only information (sensor information) from the sensor device 24 without using map information prepared in advance. The lane information is information that indicates a trajectory on which the moving body can travel. The trajectory is information that indicates the position of the moving body at multiple times.

[0029] 3, a moving object 61 generally moves along a lane (dashed line). Therefore, in predicting a trajectory using map information prepared in advance, the movement range of the moving object 61 can be narrowed down, although problems of acceleration / deceleration and lane selection arise.

[0030] An example of map information that can be used when only sensor information is used is an obstacle map that indicates the presence or absence of obstacles in a grid. Figure 4 shows an example of such an obstacle map. For example, grids where no obstacles exist are shown in white, grids where no obstacles exist are shown in black, and grids where it is unclear whether an obstacle exists or not are shown in gray. In predicting a trajectory using an obstacle map like that in Figure 4, it is clear that the mobile object 61 moves through an area where no obstacles exist (white area), but because the mobile object 61's range of movement is wide, it is not possible to narrow down the area where the mobile object 61 should pass (arrow), and the prediction becomes unstable.

[0031] Therefore, in this embodiment, a cumulative map is generated and used as map information that supplements the obstacle map. The functions of the prediction device 40 of this embodiment will be described in detail below.

[0032] Fig. 5 is a block diagram showing an example of a functional configuration of the prediction device 40 according to the present embodiment. As shown in Fig. 5, the prediction device 40 includes a mobile object information acquisition unit 101, an environment information acquisition unit 102, an environment map generation unit 103, an accumulation map generation unit 104, a prediction unit 105, and a learning unit 106.

[0033] The mobile object information acquisition unit 101 acquires mobile object information indicating the positions of one or more mobile objects including a mobile object (first mobile object) to be predicted. For example, the mobile object information acquisition unit 101 acquires the mobile object information by a sensor device 24, vehicle-to-vehicle communication, road-to-vehicle communication, etc. Road-to-vehicle communication is communication between an external device such as a roadside unit and the vehicle 10. The method of acquiring the mobile object information is not limited to this, and any method may be used.

[0034] The moving body information may further include at least one of the moving body's orientation (e.g., angle), speed, acceleration, angular velocity, angular acceleration, the variance of each of these values ​​(orientation, speed, acceleration, angular velocity and angular acceleration), direction of movement (e.g., turn signal information), and identification information (e.g., object ID).

[0035] The environmental information acquisition unit 102 acquires environmental information. For example, the environmental information acquisition unit 102 acquires environmental information using only information obtained from the sensor device 24. The environmental information is information indicating the environment around the mobile object to be predicted. For example, the environmental information includes at least one of obstacles, roads, sidewalks, curbs, signs, traffic lights, and road markings. The obstacles represent objects around the mobile object.

[0036] For example, the environmental information acquisition unit 102 detects the distance to an object around the mobile unit using a distance sensor, and acquires the position information of the object (obstacle) as environmental information. Information on roads, sidewalks, curbs, signs, traffic lights, road markings, and the like can be acquired using, for example, techniques such as object detection and semantic segmentation using images captured by a camera.

[0037] The environmental information acquisition unit 102 may acquire environmental information by combining a plurality of methods. For example, an obstacle may be detected using a distance sensor, and the specific type of object that is detected may be further detected by a technique such as object detection and semantic segmentation using an image captured by a camera.

[0038] The environment map generating unit 103 generates an environment map. The environment map is map information that shows the environment around the moving object at a reference time on a map. For example, the environment map generating unit 103 generates the environment map using moving object information and environment information at the reference time.

[0039] The reference time is, for example, the current time. The reference time may be a time prior to the current time. For example, the trajectory of a moving object predicted using a time prior to the current time as the reference time is compared with the actual trajectory (such as the trajectory up to the current time) to evaluate the prediction process. The function of the environment map generating unit 103 will be described in detail later.

[0040] The cumulative map generating unit 104 generates a cumulative map. The cumulative map is map information that shows on a map a plurality of positions indicated by mobile body information acquired at a plurality of times (first times) before a reference time. In this manner, the cumulative map is generated so that not only the positions indicated by the reference time but also the positions indicated by the mobile body information acquired before the reference time are accumulated and stored on the map.

[0041] For example, a value indicating the presence of a moving object (e.g., "1") is set to a grid corresponding to a position where a moving object was present at any of a number of times, and a value indicating the absence of a moving object (e.g., "0") is set to a grid corresponding to a position where a moving object was not present at any time.

[0042] The cumulative map may further associate at least one of the movement amount, speed, and movement direction of the moving object with each position (grid). The movement amount, speed, and movement direction may be expressed by a mixture distribution. The function of the cumulative map generating unit 104 will be described in detail later.

[0043] The prediction unit 105 predicts the future position of the moving object from the moving object information, the environment map, and the accumulation map, and outputs a predicted trajectory of the moving object. The future position may be represented by the coordinates of the position, or may be represented by, for example, the amount of movement from the current position. For example, the prediction unit 105 predicts the position of the moving object from the environment map at a reference time, the moving object information acquired at the reference time, and the accumulation map in which the moving object information up to the reference time is accumulated. The prediction unit 105 predicts the position of the moving object using, for example, a model such as a neural network that inputs the environment map, the moving object information, and the accumulation map and outputs a predicted result of the position of the moving object.

[0044] The moving body to be predicted is not limited to one, and may be multiple. The prediction unit 105 predicts and outputs a trajectory for each of the one or more moving bodies to be predicted. A prediction unit 105 may be assigned to each of the one or more moving bodies, and each prediction unit 105 may predict the trajectory of the corresponding moving body. The moving body to be predicted may be the vehicle 10 itself. For example, the moving body information acquisition unit 101 acquires moving body information of the vehicle 10 itself by vehicle-to-vehicle communication and road-to-vehicle communication, etc.

[0045] The learning unit 106 learns the model (neural network) used by the prediction unit 105. The learning unit 106 may have a function of learning the neural network (details will be described later) used by the environment map generation unit 103. Note that, when a model and neural network that have been learned in advance are used, the prediction device 40 does not need to include the learning unit 106. Details of the learning process by the learning unit 106 will be described later.

[0046] Next, an overview of the trajectory prediction process performed by the prediction device 40 will be described. Fig. 6 is a diagram for explaining the overview of the trajectory prediction process. As shown in Fig. 6, the cumulative map generating unit 104 generates a cumulative map from a plurality of pieces of moving body information acquired at a time (t=-10) before a reference time (t=0) and a plurality of pieces of moving body information acquired at the reference time. Meanwhile, the environmental map generating unit 103 generates an environmental map from environmental information acquired at the reference time.

[0047] As shown in FIG. 6, when multiple pieces of moving body information are acquired at a reference time, the prediction unit 105 predicts the future trajectory for each of the multiple moving bodies corresponding to the multiple pieces of moving body information, and outputs one or more predicted trajectories as the prediction result.

[0048] Next, a detailed description will be given of the functions of the environment map generating unit 103. Fig. 7 is a block diagram showing a detailed example of the functional configuration of the environment map generating unit 103. As shown in Fig. 7, the environment map generating unit 103 includes an obstacle map generating unit 301, an attribute map generating unit 302, and a path map generating unit 303.

[0049] The obstacle map generating unit 301 receives the environmental information acquired by the environmental information acquiring unit 102 and the mobile object information acquired by the mobile object information acquiring unit 101, and generates an obstacle map 311. The attribute map generating unit 302 generates an attribute map 312 indicating attributes of the environment around the mobile object. The route map generating unit 303 generates a route map 313 indicating routes that the mobile object may travel.

[0050] The environment map generating unit 103 generates an environment map 314 by integrating the obstacle map, the attribute map, and the route map. The environment map generating unit 103 may generate any one of the obstacle map, the attribute map, and the route map as the environment map, or may generate map information by integrating two of the obstacle map, the attribute map, and the route map as the environment map. In this case, the environment map generating unit 103 may not include components that are not necessary for generating the environment map among the obstacle map generating unit 301, the attribute map generating unit 302, and the route map generating unit 303.

[0051] Fig. 8 is a diagram showing an example of an obstacle map. Fig. 8 shows an example of an obstacle map 821 generated for a road situation as shown on the left. In the road situation shown in the example of Fig. 8, vehicles 811 and 812, which are other moving bodies, exist around a vehicle 801 corresponding to the vehicle 10.

[0052] The obstacle map 821 is map information that uses a number of grids to indicate the presence or absence of obstacles when observed from above the vehicle 801. Each grid is associated with information that indicates the presence or absence of an obstacle using a probability value between 0 and 1. In the obstacle map 821, grids where no obstacles exist are shown in white (probability value = 0), grids where no obstacles exist are shown in black (probability value = 1), and grids where it is unclear whether an obstacle exists or not are shown in gray with a density corresponding to the probability value.

[0053] For example, when a light beam (such as a laser beam) is projected from the vehicle 10, the grid corresponding to the object (moving body or obstacle) that the light beam reaches is set to 1 (obstacle present), and the grid corresponding to the area between the vehicle 10 and the object and where no object exists is set to 0 (no obstacle). If the light beam does not reach any object, the grid through which the light beam passes is set to, for example, 0.5 (unknown).

[0054] The obstacle map generating unit 301 may generate an obstacle map so that the position of an obstacle is ambiguous. For example, the presence or absence of an obstacle may be represented by a probability distribution centered on a grid where an object is present. Alternatively, the obstacle map generating unit 301 may generate an obstacle map that represents the presence or absence of only objects excluding a moving object to be predicted.

[0055] Map information such as attribute maps, route maps, environment maps, and accumulation maps (described later) are represented in a grid format similar to that of obstacle maps. The size (resolution) and shape of the grid do not need to be the same for each piece of map information, and may be different from each other. When integrating (combining) multiple pieces of map information with different grid sizes and shapes, they can be integrated after converting them to match the size and shape of the grid.

[0056] The attribute map generating unit 302 generates an attribute map indicating attributes of each grid, for example. For example, the attribute map generating unit 302 generates an attribute map indicating which of the objects a grid is, based on environmental information such as roads, sidewalks, curbs, signs, traffic lights, and road markings extracted from an image captured by a camera.

[0057] Each grid in the attribute map is represented by multiple dimensional information. Each dimension indicates an extracted attribute. The attribute may be expressed in any format. For example, the attribute may be expressed in a one-hot format in which only one dimension is 1 and the rest are 0. For each of the multiple dimensions, the likelihood of the corresponding attribute obtained by semantic segmentation may be set.

[0058] The route map generating unit 303 generates a route map using, for example, a neural network. The route map is map information including either a reward map or a policy map. The reward map is map information indicating rewards for actions that a moving object can take. For example, the reward map is map information in a format in which a scalar value indicating a reward for the existence of a moving object is set for each grid. The policy map is map information indicating policies that a moving object can take based on rewards.

[0059] The reward map is calculated by a neural network that receives at least one of the obstacle map and the attribute map as input. For example, the neural network includes a convolutional neural network (CNN) that receives at least one of the obstacle map and the attribute map as input and outputs a reward map.

[0060] The neural network for generating the reward map is trained according to inverse reinforcement learning and imitation learning. For example, the path of the taught moving object is used as the expert of the behavior required for the training data. The path does not represent the amount of movement at each time step, but represents time-series data representing the order in which four actions were selected on a grid map: one square up, one square down, one square left, and one square right. Stopping may or may not be added to the behavior.

[0061] The policy map can be generated based on the reward map. The policy map is a route in which actions are selected so that the reward obtained is high, starting from the position of each moving body at a reference time. The policy map may indicate one or more policies that each moving body can take. Since the policy map starts from the position of each moving body, it is generated for each moving body. Therefore, a route map based on the policy map is also generated for each moving body.

[0062] The environment map generating unit 103 generates an environment map by linking, in the channel direction, at least one piece of map information among the obstacle map, the attribute map, and the route map generated as described above.

[0063] Each channel corresponds to a piece of map information to be linked. When all of the obstacle map, attribute map, and route map are linked, the number of channels is three. A value of the map information corresponding to each channel is set in each grid of the environment map. When the grid resolutions of the map information are different, the environment map generating unit 103 links each piece of map information after converting them so that the resolutions match.

[0064] When a route map is generated for each moving body, the environment map generating unit 103 generates an environment map for each moving body. The environment map generated for each moving body is sent to the predicting unit 105. When the predicting unit 105 is assigned to each of one or more moving bodies, the corresponding environment map is sent to the predicting unit 105 corresponding to each moving body.

[0065] Next, the details of the function of the cumulative map generating unit 104 will be described. Fig. 9 is a diagram showing an example of a cumulative map generated by the cumulative map generating unit 104. Fig. 9 shows examples of cumulative maps 921, 922, and 923 generated at times t=-20, -10, and 0, respectively. In the upper part of the figure, examples of obstacle maps 911, 912, and 913 generated at each time are shown to explain the road conditions at each time. In the cumulative map of Fig. 9, "1" is set to a grid where a moving object exists, and "0" is set to a grid where a moving object does not exist.

[0066] At t=-20, another moving object, vehicle 901, is observed for the first time. At this time, the values ​​of each grid in the cumulative map are all 0.

[0067] At t=-10, vehicle 901 is traveling straight and another moving body, vehicle 902, is about to turn left. Accumulative map generating unit 104 sets “1” to the grids on the cumulative map corresponding to the trajectories of vehicle 901 and vehicle 902.

[0068] Similarly, when t=0, the cumulative map generating unit 104 sets “1” to the grid on the cumulative map that corresponds to the trajectory of the vehicle 902 .

[0069] The cumulative map generated in this manner is utilized for prediction of yet another moving body, vehicle 903, at the reference time t=0. For example, a grid in which "1" is set in the cumulative map is highly likely to be an area in which vehicle travel is recommended. Therefore, by using a cumulative map including such information, it is possible to improve the accuracy of trajectory prediction of the moving body (vehicle 903).

[0070] In the accumulation map of Fig. 9, information of one channel indicating whether or not a moving object was present is set in a grid. Information of multiple channels may be set in each grid. Each channel stores, for example, any one of the movement amount, speed, and direction (movement direction) of the moving object.

[0071] The cumulative map may accumulate multidimensional information or information expressed as a mixed distribution for each channel (movement amount, speed, direction, etc.). For example, since moving objects moving in various directions, such as going straight and turning right and left, intersect near the center of an intersection, it is difficult to hold information such as speed and direction in one dimension. Therefore, the cumulative map generating unit 104 compresses information observed at multiple times into multidimensional information and accumulates it. The cumulative map generating unit 104 may compress information observed at multiple times by expressing it as a multidimensional mixed distribution.

[0072] For example, principal component analysis, EM algorithm, and recurrent neural network (RNN) can be applied to compress the information. For example, the cumulative map generating unit 104 inputs the information already set in the grid and the acquired mobile object information to a recurrent neural network, and sets multidimensional information with a predetermined number of dimensions, including a mixture distribution output from the recurrent neural network, as new information for the grid. By compressing and accumulating the information, it is possible to reduce the storage capacity required for the cumulative map.

[0073] The cumulative map can only be used in the same area. For this reason, the cumulative map is generated by accumulating information in a predetermined range centered on the vehicle 10. When the vehicle 10 moves, the cumulative map is moved and rotated according to the amount of movement, and the accumulated information is maintained. Similarly to the cumulative map, the environmental map is also generated, for example, centered on the vehicle 10. When a sensor is provided in an external device (for example, a roadside unit), the environmental map and the cumulative map are generated centered on the mounting position of the sensor.

[0074] Next, the details of the function of the prediction unit 105 will be described. The behavior of the prediction unit 105 differs between inference and learning. FIG. 10 is a block diagram showing a detailed functional configuration example of the prediction unit 105 at the time of inference. As shown in FIG. 10, the prediction unit 105 includes a time-series feature extraction unit 501, a spatial feature extraction unit 502, a time-space feature integration unit 503, a sampling unit 504, and a trajectory generation unit 505.

[0075] The time series feature extraction unit 501 extracts and outputs time series features from the moving body information. The time series feature extraction unit 501 inputs data (input data) obtained over one or more time periods as one-dimensional vectors including at least one of position, angle, velocity, angular velocity, acceleration, and angular acceleration, which is the moving body information obtained by the moving body information acquisition unit 101. The time series features output by the time series feature extraction unit 501 are information that characterizes the amount of movement change of the moving body in a time series.

[0076] The time-series feature extraction unit 501 includes, for example, a recurrent neural network and a fully connected layer, and repeatedly inputs the above-mentioned input data. As the type of recurrent neural network, a simple recurrent network such as a simple RNN, a long short term memory (LSTM), or a gated recurrent unit (GRU) may be used.

[0077] The spatial feature extraction unit 502 receives as input data the position and angle of the moving object information acquired by the moving object information acquisition unit 101, the environment map generated by the environment map generation unit 103, and the cumulative map generated by the cumulative map generation unit 104, and outputs spatial features. The spatial features are information that characterizes the surrounding information of the moving object. For example, the spatial feature extraction unit 502 obtains spatial features for the input data using a neural network that receives the input data and outputs spatial features.

[0078] The coordinate system of the environmental map and the cumulative map is centered on the mounting position of the sensor of the vehicle 10 or the external device. Therefore, when using it as input data for a neural network, the coordinate system needs to be normalized for each moving body. That is, the spatial feature extraction unit 502 cuts out and rotates the environmental map and the cumulative map using the position and angle of the moving body information at the reference time.

[0079] Fig. 11 is a diagram showing an example of normalization of map information for each moving object. In Fig. 11, vehicles 901 and 902 are centered and cropped and rotated into a rectangle of a predetermined size so that the traveling direction faces upward. Rectangles 1101 and 1102 are examples of map information normalized for vehicles 901 and 902, respectively.

[0080] When an area not included in the map information before normalization falls within the range of the map information after normalization, the spatial feature extraction unit 502 may set a predetermined value or a value estimated from the values ​​of adjacent areas, etc. For example, "0", "0.5 (unknown in obstacle map)", or "1" is set.

[0081] The above normalization method is an example, and is not limited to this. For example, a method of cutting out a rectangle only in the front range of the moving object, a method of not performing rotation processing, or the like may be adopted as the normalization method.

[0082] The spatial feature extraction unit 502 inputs the normalized environment map and the cumulative map to the neural network. The neural network is, for example, configured as a CNN, and outputs spatial features contracted to a one-dimensional vector. The neural network may be configured to give weights (attention) to each grid of the environment map based on the values ​​obtained in the cumulative map. For example, if the speed of a moving object is set to each grid of the cumulative map, a weight whose value varies according to the speed is assigned to the corresponding grid of the environment map. The neural network may also be configured to handle the environment map and the cumulative map by linking them in the channel direction.

[0083] The environment map may include an attribute map detected by semantic segmentation and a route map calculated using a trained neural network. Therefore, the environment map may contain erroneous information. In contrast, the cumulative map is generated from actually observed information, so there is a small possibility that it may contain errors. Therefore, it is possible to improve the prediction accuracy by providing weights to each grid of the environment map based on the cumulative map, or by linking the environment map and the cumulative map.

[0084] The spatiotemporal feature integration unit 503 receives the time series features extracted by the time series feature extraction unit 501 and the spatial features extracted by the spatial feature extraction unit 502 as input data, and outputs spatiotemporal features corresponding to the feature amount obtained by integrating the two. For example, the spatiotemporal feature integration unit 503 obtains spatiotemporal features using a neural network that receives the input data and outputs the spatiotemporal features. Since the time series features and the spatial features are both one-dimensional vectors, a neural network including a fully connected layer that receives input data in which the time series features and the spatial features are linked in the dimensional direction and outputs the spatiotemporal features can be used.

[0085] The sampling unit 504 performs random sampling within a one or more dimensional multidimensional normal distribution to generate a latent variable of the trajectory. The latent variable is a variable that expresses a series of movements of the trajectory within a multidimensional normal distribution. The latent variable can be interpreted as being expressed by a multidimensional normal distribution that expresses the characteristics of the trajectory of the moving object. When the trajectory generation unit 505 outputs multiple trajectories, the sampling unit 504 samples multiple latent variables. The multidimensional normal distribution that expresses the latent variable is learned together with the learning of the neural network used in the prediction unit 105. The details of the learning method of the multidimensional normal distribution will be described later.

[0086] The trajectory generation unit 505 generates a predicted trajectory using the sampled latent variables as input. The number of times random sampling is performed by the sampling unit 504 (the number of latent variables) is the number of predicted trajectories output by the trajectory generation unit 505.

[0087] The trajectory generating unit 505 receives the spatiotemporal features obtained by the spatiotemporal feature integrating unit 503 and the latent variables obtained by the sampling unit 504 as input data, and outputs a predicted trajectory. The trajectory generating unit 505 generates a predicted trajectory for each latent variable. A trajectory generating unit 505 may be assigned to each of one or more latent variables, and each trajectory generating unit 505 may perform trajectory prediction for the corresponding latent variable.

[0088] For example, the trajectory generation unit 505 obtains a predicted trajectory using a neural network that inputs the above input data and outputs a predicted trajectory. Since both the spatiotemporal features and the latent variables are one-dimensional vectors, it is possible to use a neural network that inputs input data that links these in the dimensional direction and outputs a predicted trajectory.

[0089] The neural network used by the trajectory generation unit 505 includes, for example, a recurrent neural network and a fully connected layer. The recurrent neural network performs repeated calculations for each time step until a specified prediction time is reached. The input of the recurrent neural network at each time step is the same input data that combines spatiotemporal features and latent variables. The internal variables in the recurrent neural network are successively updated by performing repeated calculations.

[0090] The output from the fully connected layer of the recurrent neural network at each time step is a coordinate value representing a future position, or information representing a distribution of future positions. The coordinate value representing a position is, for example, two-dimensional information of an x ​​coordinate value and a y coordinate value. The information representing the distribution of positions is, for example, five-dimensional information including a two-dimensional coordinate value representing a position peak, a two-dimensional value representing a variance, and a one-dimensional value representing a correlation coefficient.

[0091] The calculation is repeated for each time step until the specified prediction time is reached, and the set of future predicted positions becomes one predicted trajectory. The trajectory generation unit 505 performs such calculation for each latent variable, and generates predicted trajectories for the number of latent variables.

[0092] Although the functions of the prediction unit 105 have been described above as being divided into the various units (time-series feature extraction unit 501, spatial feature extraction unit 502, time-space feature integration unit 503, sampling unit 504, and trajectory generation unit 505), some or all of these functions may be integrated. For example, a single neural network obtained by integrating the neural networks used by each unit may be used as the neural network for trajectory prediction by the prediction unit 105.

[0093] Next, a learning method by the learning unit 106 of the neural network used by the prediction unit 105 will be described. Fig. 12 is a block diagram showing a detailed functional configuration example of the prediction unit (hereinafter, referred to as the prediction unit 105b) during learning. As shown in Fig. 12, the prediction unit 105b includes time-series feature extraction units 501 and 501b, a spatial feature extraction unit 502, a time-space feature integration unit 503, a latent feature learning unit 504b, and a trajectory generation unit 505. The same components as those during inference are denoted by the same reference numerals, and description thereof will be omitted.

[0094] The time series feature extraction unit 501b inputs the true value of the moving body information (the true value of the trajectory of the moving body) and outputs the time series feature for the true value of the moving body information. The true value of the moving body information is data of a one-dimensional vector including at least one of the position, angle, speed, angular velocity, acceleration, and angular acceleration of the moving body at a future time, acquired for one or more time periods. For example, after sequentially inputting the moving body information to the time series feature extraction unit 501, the true value is input to the current neural network of the time series feature extraction unit 501b having the same weight as the recurrent neural network of the time series feature extraction unit 501, whereby the time series feature of the trajectory corresponding to the true value can be extracted.

[0095] The latent feature learning unit 504b uses a neural network to learn a multidimensional normal distribution of one or more dimensions so as to approximate the time series characteristics of the trajectory corresponding to the true value. The neural network inputs the time series characteristics of the trajectory corresponding to the true value to a fully connected layer, and outputs them by dividing them into one or more dimensional means and variances. A loss indicating the error between the distribution represented by the mean and variance and the multidimensional normal distribution is added to the learning loss function. This makes it possible to obtain a multidimensional normal distribution that approximates the time series characteristics of the trajectory corresponding to the true value. For example, the KL divergence distance can be used as the loss indicating the error between the distributions.

[0096] The latent feature learning unit 504b uses, for example, the reparameterization trick to generate latent features to be input to the trajectory generation unit 505. When the mean generated by the latent feature learning unit 504b is μ and the variance is Σ, the latent variable can be expressed by the following formula (1). μ = √Σ × N(0,I) (1)

[0097] Here, I is the identity matrix. N(0,I) represents random sampling within a multidimensional normal distribution with mean μ = 0 and variance Σ = 1. This configuration enables learning by backpropagation of latent variables.

[0098] The learning unit 106 performs learning so as to minimize a loss function. The loss function includes a loss (loss of predicted trajectory) indicating an error between the predicted trajectory generated by the trajectory generation unit 505 and the true trajectory, in addition to the loss between the distributions (such as the KL divergence distance).

[0099] For example, when the output of the trajectory generating unit 505 is information representing a distribution of positions (such as a two-dimensional normal distribution), the loss of the predicted trajectory is designed so that the probability that a position that is a true value (true value position) exists is maximized (minimized by adding the negative value) in a distribution (such as a two-dimensional normal distribution) represented by the future position output at each time step, variance, and correlation coefficient. Also, when the output of the trajectory generating unit 505 is a coordinate value representing a position, the loss of the predicted trajectory is designed so as to minimize the absolute error or squared error between the output coordinate value and the true value position.

[0100] Next, a flow of prediction processing by the prediction device 40 according to this embodiment configured as described above will be described. Fig. 13 is a flowchart showing an example of prediction processing in this embodiment.

[0101] The mobile object information acquisition unit 101 acquires mobile object information from the sensor device 24 or the like (step S101). The cumulative map generation unit 104 generates a cumulative map using the acquired mobile object information (step S102). The processes of steps S101 and S102 are executed at multiple times until a reference time is reached.

[0102] The environmental information acquisition unit 102 acquires environmental information using information obtained from the sensor device 24 (step S103). The environmental map generation unit 103 generates an environmental map using the mobile object information and the environmental information acquired at a reference time (step S104).

[0103] The prediction unit 105 predicts the movement of the moving object by using the moving object information acquired at the reference time, the cumulative map generated up to the reference time, and the environmental map generated at the reference time (step S105). The prediction unit 105 outputs the predicted trajectory obtained by the prediction (step S106). After that, the process returns to step S101 and is repeated.

[0104] Next, a flow of the learning process by the prediction device 40 according to this embodiment configured as above will be described below. Fig. 14 is a flowchart showing an example of the learning process in this embodiment.

[0105] The prediction unit 105b (time-series feature extraction unit 501, spatial feature extraction unit 502, and spatiotemporal feature integration unit 503) used during learning calculates spatiotemporal features using the moving object information, the accumulation map, and the environment map (step S201). The prediction unit 105b (time-series feature extraction unit 501b) also calculates time-series features using the true value of the moving object information (step S202).

[0106] The prediction unit 105b (latent feature learning unit 504b) calculates latent features from the time-series features calculated in step S202 (step S203). The prediction unit 105b predicts a trajectory from the spatiotemporal features calculated in step S201 and the latent features calculated in step S203 (step S204).

[0107] The learning unit 106 executes a learning process so as to minimize a loss function including a loss indicating an error between the predicted trajectory and the true value of the trajectory, and a loss indicating an error between distributions (step S205). The learning unit 106 determines whether learning has ended (step S206). For example, the learning unit 106 determines whether learning has ended based on whether the improvement in the loss has become smaller than a threshold value, whether the number of learning times has reached an upper limit, and so on.

[0108] If the learning is not completed (step S206: No), the process returns to step S201, and the process is repeated for new learning data. If it is determined that the learning is completed (step S206: Yes), the learning process ends.

[0109] In this manner, in this embodiment, a cumulative map showing areas where multiple moving objects are likely to pass is generated only from information obtained from the sensor, and the trajectory of the moving object is predicted using the generated cumulative map. This makes it possible to more accurately predict the future position of the moving object without using a high-precision map, which requires a large processing load for generation.

[0110] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0111] 10 Vehicles 12 Processing equipment 21 Storage device 22 Input Devices 23 Display device 24 Sensor device 25 Communication Equipment 26 Vehicle control device 30 Information processing device 31 Memory 32 Hardware Processors 40 Prediction Device 101 Mobile information acquisition unit 102 Environmental Information Acquisition Department 103 Environment Map Generation Unit 104 Accumulative map generator 105, 105b Prediction Section 106 Learning Department 301 Obstacle map generation unit 302 Attribute Map Generator 303 Route map generation unit 501, 501b Time series feature extraction unit 502 Spatial Feature Extraction Unit 503 Spatio-temporal Feature Integration Unit 504 Sampling section 504b Latent feature learning unit 505 Trajectory generation part

Claims

1. a moving object information acquiring unit that acquires moving object information indicating positions of one or more moving objects that move along lanes provided on a road, including a first moving object that is a prediction target; a cumulative map generating unit that generates cumulative map information that shows, on a map, a plurality of positions indicated by the moving object information acquired at a plurality of first times before a reference time; a prediction unit that predicts a position of the first moving body at a second time after the reference time based on environmental map information that shows an environment around the first moving body at the reference time on a map, the moving body information acquired at the reference time, and the cumulative map information, the environmental map information includes route map information indicating a route on which the first moving object may travel; the route map information includes any one of reward map information indicating rewards for actions that the first moving body can take, and policy map information indicating policies that the first moving body can take based on the rewards, the rewards being calculated by a neural network that receives as input at least one of obstacle map information indicating the presence or absence of obstacles and attribute map information indicating attributes of the environment; Prediction device.

2. The environment map information does not include information indicating the lane. The prediction device according to claim 1 .

3. the cumulative map generation unit generates the cumulative map information in which at least one of a movement amount of the moving body, a speed of the moving body, and a movement direction of the moving body is associated with each of the plurality of positions indicated by the moving body information acquired at the plurality of first times; The prediction device according to claim 1 .

4. the cumulative map generation unit generates the cumulative map information in which at least one of a mixed distribution of movement amounts of the plurality of moving bodies, a mixed distribution of velocities of the plurality of moving bodies, and a mixed distribution of movement directions of the plurality of moving bodies is associated with each of the plurality of positions indicated by the moving body information acquired at the plurality of first times; The prediction device according to claim 1 .

5. The moving body information further includes at least one of a direction of the moving body, a speed of the moving body, an acceleration of the moving body, an angular velocity of the moving body, an angular acceleration of the moving body, a moving direction of the moving body, and identification information of the moving body. The prediction device according to claim 1 .

6. An environmental information acquisition unit that acquires environmental information indicating the environment; and an environment map generating unit that generates the environment map information based on the environment information and the moving object information. The prediction device according to claim 1 .

7. The environment map information further includes at least one of the obstacle map information and the attribute map information. The prediction device according to claim 6.

8. The environmental information includes at least one of obstacles, roads, sidewalks, curbs, signs, traffic lights, and road markings; The prediction device according to claim 6 .

9. the prediction unit predicts a position of the first moving object by using a neural network to which the environment map information, the moving object information, and the cumulative map information are input; The prediction device according to claim 1 .

10. The neural network predicts one or more positions according to one or more variables sampled based on a multidimensional normal distribution that represents a characteristic of a trajectory of a moving object. The prediction device according to claim 9 .

11. the prediction unit predicts a position of the first moving object at the second time by using the environment map information to which a weight corresponding to a value set in the cumulative map information is assigned. The prediction device according to claim 1 .

12. a moving object information acquiring step of acquiring moving object information indicating positions of one or more moving objects moving along lanes provided on a road, including a first moving object to be predicted; a cumulative map generating step of generating cumulative map information showing on a map a plurality of positions indicated by the moving object information acquired at a plurality of first times before a reference time; a prediction step of predicting a position of the first moving body at a second time after the reference time based on environmental map information showing an environment around the first moving body at the reference time on a map, the moving body information acquired at the reference time, and the cumulative map information, the environmental map information includes route map information indicating a route on which the first moving object may travel; the route map information includes any one of reward map information indicating rewards for actions that the first moving body can take, and policy map information indicating policies that the first moving body can take based on the rewards, the rewards being calculated by a neural network that receives as input at least one of obstacle map information indicating the presence or absence of obstacles and attribute map information indicating attributes of the environment; Forecasting methods.

13. On the computer, a moving object information acquiring step of acquiring moving object information indicating positions of one or more moving objects moving along lanes provided on a road, including a first moving object to be predicted; a cumulative map generating step of generating cumulative map information showing on a map a plurality of positions indicated by the moving object information acquired at a plurality of first times before a reference time; a prediction step of predicting a position of the first moving body at a second time after the reference time based on environmental map information showing an environment around the first moving body at the reference time on a map, the moving body information acquired at the reference time, and the cumulative map information; the environmental map information includes route map information indicating a route on which the first moving object may travel; the route map information includes any one of reward map information indicating rewards for actions that the first moving body can take, and policy map information indicating policies that the first moving body can take based on the rewards, the rewards being calculated by a neural network that receives as input at least one of obstacle map information indicating the presence or absence of obstacles and attribute map information indicating attributes of the environment; program.

14. A vehicle control system for controlling a vehicle, A prediction device that predicts a position of a first moving object to be predicted; a vehicle control device that controls a drive mechanism for driving the vehicle based on the predicted position; Equipped with The prediction device comprises: a moving object information acquiring unit that acquires moving object information indicating positions of one or more moving objects that move along lanes provided on a road, including a first moving object that is a prediction target; a cumulative map generating unit that generates cumulative map information that shows, on a map, a plurality of positions indicated by the moving object information acquired at a plurality of first times before a reference time; a prediction unit that predicts a position of the first moving body at a second time after the reference time based on environmental map information that shows an environment around the first moving body at the reference time on a map, the moving body information acquired at the reference time, and the cumulative map information, the environmental map information includes route map information indicating a route on which the first moving object may travel; the route map information includes any one of reward map information indicating rewards for actions that the first moving body can take, and policy map information indicating policies that the first moving body can take based on the rewards, the rewards being calculated by a neural network that receives as input at least one of obstacle map information indicating the presence or absence of obstacles and attribute map information indicating attributes of the environment; Vehicle control system.

Images