Device for predicting the behavior of a moving body and method for predicting the behavior of a moving body

Abstract

Device for predicting the behavior of a moving body, comprising: a first behavior prediction unit (203) that outputs a first predicted behavior (204) of a moving body based on a prediction result of a behavior of the moving body detectable by a vehicle and a detection result of a behavior of the moving body after a prediction time has elapsed; and a second behavior prediction unit (207) that outputs a second predicted behavior (208) of the moving body detectable by the vehicle based on the behavior of the vehicle.

Description

Technical field

[0001] The present invention relates to a device for predicting the behavior of a moving body and a method for predicting the behavior of a moving body, which can be applied to the automatic driving of a motor vehicle or the like. State of the art

[0002] To realize automated driving of motor vehicles, a sensing technology is being developed that captures environmental conditions using in-vehicle cameras, etc.; a recognition technology that detects the state of the vehicle and its surroundings based on the captured data; and a control technology to regulate driving speed and steering angle based on the recognition information about the state of the vehicle and its surroundings. The recognition technology requires a predictive technology that detects an object or moving body existing around the vehicle and accurately predicts its future position.

[0003] Various factors, such as the interaction between moving bodies and their surroundings, influence the future behavior of moving bodies like pedestrians and vehicles. Since it is difficult to articulate all of these effects, the effects of each factor can be treated as a black box using machine learning.

[0004] PTL 1, for example, discusses a mechanism for predicting the future position of a moving body using regression analysis. Supervised learning is generally used for the prediction problem. List of prior art patent literature

[0005] PTL 1: JP 2013-196601 A Summary of the invention: Technical problem

[0006] The predictor obtained through supervised learning is strong for a frequent pattern but exhibits poor predictive accuracy for a rare pattern. On the other hand, in the case of autonomous driving, it is necessary to consider infrequent actions such as a pedestrian suddenly stepping out, another vehicle suddenly accelerating / decelerating, and lane changes for safety reasons. Therefore, it is difficult to achieve safe autonomous driving using predictive techniques based on simple supervised learning.

[0007] Furthermore, in supervised learning, when only rare pattern data such as sudden acceleration / deceleration, lane changes, etc. are used for learning, only a prediction of a rare pattern is made, which hinders normal safe driving.

[0008] The invention was carried out in light of the above circumstances, and one object of the invention is to provide a device for predicting the behavior of a moving body and a method for predicting the behavior of a moving body that can improve the accuracy of predicting a rare behavior of the moving body without reducing the accuracy of predicting the behavior of the moving body that occurs frequently. Solution to the problem

[0009] To solve the above problem, the device for predicting the behavior of a moving body according to a first aspect comprises a first behavior prediction unit that outputs a first predicted behavior of a moving body based on a prediction result of a behavior of the moving body detectable by a vehicle and a detection result of a behavior of the moving body after a prediction time has elapsed, and a second behavior prediction unit that outputs a second predicted behavior of the moving body detectable by the vehicle based on the behavior of the vehicle. Advantageous effects of the invention

[0010] According to the invention, it is possible to improve the accuracy of the prediction of rarely occurring behavior of moving bodies without reducing the accuracy of the prediction of commonly occurring behavior of moving bodies. List of characters [ Fig. 1] Fig. Figure 1 is a schematic diagram representing an example of a driving environment of a motor vehicle to which a device for predicting the behavior of a moving body according to a first embodiment is applied. [ Fig. 2] Fig. Figure 2 is a block diagram that represents a configuration of the device for predicting the behavior of a moving body according to the first embodiment. [ Fig. 3] Fig. Figure 3 is a block diagram showing the configuration of a detection unit in Fig. 2 represents. [ Fig. 4] Fig. 4 is a diagram that shows a configuration example of map information in Fig. 3 represents. [ Fig. 5] Fig. Figure 5 is a block diagram that represents a configuration of a behavior prediction unit used in the device for predicting the behavior of a moving body according to the embodiment. [ Fig. 6] Fig. 6 is a block diagram showing a configuration of a control unit in Fig. 2 represents. [ Fig. 7] Fig. 7(a) is a schematic diagram illustrating an evaluation procedure of a driving assessment unit in Fig. 2 represents, Fig. 7(b) is a diagram that shows an example of a data map in Fig. 5 represents, and Fig. 7(c) is a diagram that shows an example of behavioral data from a future time. Fig. 5 represents. [ Fig. 8] Fig. Figure 8 is a diagram that illustrates a first prediction behavior and a second prediction behavior, as displayed by the device for predicting the behavior of a moving body. Fig. 2 can be predicted. [ Fig. 9] Fig. Figure 9 is a block diagram that represents a configuration of a device for predicting the behavior of a moving body according to a second embodiment. [ Fig. 10] Fig. Figure 10 is a block diagram representing a hardware configuration of a device for predicting the behavior of a moving body according to a third embodiment. Description of embodiments

[0011] Embodiments are described with reference to the drawings. Furthermore, the embodiments described below do not limit the scope of the invention. Not all elements and combinations thereof described in the embodiments are essential for the solution of the invention. (First embodiment)

[0012] Fig. Figure 1 is a schematic diagram representing an example of a driving environment of a motor vehicle to which a device for predicting the behavior of a moving body according to the first embodiment is applied.

[0013] In Fig. 1. It is assumed that a private vehicle 101 on a street 100 drives and other vehicles 102 and 103 in front of the own vehicle 101 drive. The other vehicles 102 and 103 are vehicles other than the owner's own vehicle 101It is assumed that a pedestrian 104 next to the road 100 running.

[0014] The private vehicle 101 is equipped with a device 10 to predict the behavior of a moving body, a sensor 20 and a display unit 30 equipped. The device 10 To predict the behavior of a moving body, a future position of a moving body, such as other vehicles, is given. 102 and 103 , of the pedestrian 104 and the motorcycle beforehand (may be referred to below as predicted behavior). The sensor 20 detects a condition of the road 100 and the moving body around the vehicle 101 As a sensor 20For example, a camera, radar, lidar (rider), sonar, GPS (global positioning system), and car navigation system can be used. The display unit 30 shows the predicted behavior that is produced by the device 10 The system is used to predict the behavior of a moving body. This predicted behavior can be displayed by overlaying it onto the image in front of the vehicle. 101 is superimposed, which is caused by the sensor 20 is recorded, or can be displayed on the windshield of the vehicle. 101 will be displayed.

[0015] For example, if the other vehicles 102 and 103 and the pedestrian 104 each along routes K2 until K4 The device can move. 10 To predict the behavior of a moving body, predict the position where the other vehicles are. 102 and103 and the pedestrian 104 probably be located. The private vehicle 101 During automatic driving, it can adjust the steering angle and speed to avoid a collision with a moving object, such as other vehicles. 102 and 103 and the pedestrian 104 sudden steering, sudden deceleration, sudden acceleration and sudden stop of the vehicle 101 to prevent, based on the prediction of the behavior of the moving body by the device 10 to predict the behavior of a moving body.

[0016] The behavior of the moving body, such as other vehicles 102 and 103 , of the pedestrian 104The behavior of a car or a two-wheeled vehicle changes according to its surroundings. For example, the vehicle's handling changes on a highway, a main road, and a secondary road. Furthermore, the behavior of the moving body also changes depending on the number of other moving bodies nearby. For instance, the vehicle's behavior changes significantly on a highway with no other moving bodies, a busy expressway, a crowded shopping street, and so on. Therefore, for safe autonomous driving, it is necessary to predict the future behavior of the moving body, taking into account road information, interactions with surrounding objects, and other factors.

[0017] The behavior of the vehicle or moving body includes a common pattern that occurs frequently and a rare pattern that occurs infrequently. The common pattern includes the normal driving of the other vehicles. 102 and 103 along the road 100 and the walking of the pedestrian 104 along the road 100 The rare pattern includes the pedestrian protruding. 104 onto the street 100 , crossing the street 100 , a sudden acceleration / deceleration of the other vehicles 102 and 103 , changing the course and the like.

[0018] In order to accommodate both the common and the rare pattern, the device provides 10To predict the behavior of a moving body, a first predictive behavior of the moving body is based on a predictive result of the behavior of the moving body around the own vehicle. 101 and a detection result of the behavior of the moving body after a prediction period. Furthermore, the device outputs 10 To predict the behavior of a moving body, a second predictive behavior of the moving body is required, which is derived from the vehicle itself. 101 This can be seen based on the behavior of the vehicle itself. 101 The first predictive behavior can be predicted from the frequent pattern. The second predictive behavior can be predicted from the rare pattern.

[0019] At this stage, it is difficult to formulate all the factors that affect the future behavior of the moving body, such as route information and interaction with surrounding objects. Therefore, by treating the influence of each factor as a black box through machine learning, it is possible to predict the future behavior of the moving body, taking into account route information, interaction with surrounding objects, and the like.

[0020] The frequent pattern is predicted through supervised learning. Here, the future position and speed of the object detected by the sensor are determined. 20 is recognized, the one on the own vehicle 101The predicted position and velocity are used as the initial predictive behavior. Afterward, the learning process is carried out in such a way that the difference between the position and velocity of the same object observed after a predetermined prediction time and the predicted future position and velocity becomes small.

[0021] The rare pattern is predicted through reinforcement learning, and the predicted future position and speed are used as a second predictive behavior. Based on the first predictive behavior through supervised learning and the second predictive behavior through reinforcement learning, it is determined whether the vehicle will 101 It can be driven safely if the vehicle is in its own possession. 101 The second predicted behavior is modified through reinforcement learning to make driving safer.

[0022] When predicting behavior using supervised learning, it is necessary to perform accurate behavior prediction for more data, so that the prediction accuracy for the common pattern is slightly improved.

[0023] Behavior prediction based on reinforcement learning requires a focus on factors that influence the control of the self-driving vehicle. 101 to make it uncertain, so that the prediction accuracy for the rare pattern, which is dangerous behavior, can be easily improved.

[0024] As described above, in the embodiment described above, by combining supervised learning and reinforcement learning, it is possible to predict the behavior of the moving body, in which both the frequent and the rare patterns are reflected, and to control the self-driving vehicle more safely.

[0025] The device for predicting the behavior of a moving body according to the embodiment is described in detail below.

[0026] Fig. Figure 2 is a block diagram illustrating a configuration of the device for predicting the behavior of a moving body according to the first embodiment. Fig. 2 includes the device 10 A recognition unit is used to predict the behavior of a moving body. 202 , a first behavioral prediction unit 203 , a prediction error calculation unit 205 , a first parameter update magnitude calculation unit 206 , a second behavioral prediction unit 207 , a control unit 209 , a driving assessment unit 210 , a reward generation unit 211 and a second parameter update magnitude calculation unit 212 .

[0027] Here, the first behavioral prediction unit can be used. 203 a first predictive behavior 204 To learn in order to minimize the error between the predicted behavior of the moving body and the actual detection of that behavior after the prediction time has elapsed. The second behavior prediction unit 207 can a future second predictive behavior 208 of the moving body around the vehicle 101 learn so that the own vehicle 101 does not engage in unsafe driving.

[0028] At this time, the first behavioral prediction unit 203 and the second behavioral prediction unit 207 the first predictive behavior 204 or the second predictive behavior 208 using the data provided by the recognition unit 202 identified result.

[0029] If the first prediction behavior 204The first behavioral prediction unit also learns the frequent pattern. 203 the first predictive behavior 204 through supervised learning, so that the own vehicle 101 can drive safely. If the second prediction behavior 208 The rare pattern is learned by the second behavioral prediction unit. 207 the second predictive behavior 208 through reinforcement learning, so that the own vehicle 101 can drive safely. Furthermore, the second predictive behavior 208 the same form as the first prediction behavior 204 assume. At this time, the configuration of the second behavioral prediction unit can be assumed. 207 the same as the configuration of the first behavioral prediction unit 203 be. Furthermore, the second behavioral prediction unit can 207 parameters with the first behavioral prediction unit 203 split.

[0030] Sensor data 201are data from the sensor 20 will be obtained, which is attached to the owner's vehicle 101 is attached. The recognition unit 202 detects other nearby vehicles and pedestrians as a result of processing sensor data 201 It receives map data, street attribute information, destination information, and the like. It also recognizes information required for behavioral prediction by the predictive model.

[0031] Fig. Figure 3 is a block diagram showing a configuration of the detection unit in Fig. 2 represents.

[0032] In Fig. 3 detects a recognition unit 202 a surrounding object and surrounding environment of the vehicle 101 based on the sensor data 201 At this time, the sensor data can be accessed 201It contains a stereo camera image and time-series data obtained from the vehicle's speed, yaw rate, GPS, and similar parameters. The detection unit 202 includes a stereo comparison unit 303 , an object recognition unit 305 , a position calculation unit 307 and an object tracking unit 311 .

[0033] The stereo comparison unit 303 creates a parallax image 304 based on the right camera image 301 and the left camera image 302 . Stereo comparison can be performed using a convolutional neural network (CNN), a block comparison method, or similar techniques.

[0034] The object recognition unit 305 Performs image processing of the left camera image 302 through and generates the object recognition result 306by recognizing an object appearing in the image. Although the example for performing object recognition processing in the configuration of Fig. 3 on the left camera image 302 As shown, object recognition processing can also be performed on the right camera image. 301 This is carried out. Here, object recognition processing is performed by the object recognition unit. 305 Detection of a moving body and semantic segmentation.

[0035] The detection of a moving object can be performed using a Faster R CNN or a CNN technique called Single Shot Multibox Detector (SSD). These are methods for determining the position and type of a target in an image. Regarding the target's position, a rectangular area containing the target in the image is output. Regarding the target's type, a class of the target, such as a person or a vehicle, contained within the rectangular area is also output for each detected rectangular area. Multiple rectangular areas can be extracted from a single image. Furthermore, Faster R CNN and SSD are examples of moving object detection and can be replaced by another method capable of detecting an object in the image.Instead of object detection, a method called instance segmentation can also be used to identify a pixel region in which each detection target is reflected for each detection target in an image. For instance segmentation, a method such as Mask R-CNN is used, but other instance segmentation methods besides Mask R-CNN can be employed.

[0036] Semantic segmentation can be performed using a CNN technique called ResNet or U-Net. Semantic segmentation is a technique for identifying which class of object each pixel in an image represents. The class identified through semantic segmentation can include not only moving bodies such as people and vehicles, but also terrain information such as roads, sidewalks, white lines, buildings, obstacles, and three-dimensional objects. Furthermore, ResNet and U-Net are examples of semantic segmentation.

[0037] The position calculation unit 307 receives the class information of the object recognition result 306 based on the parallax image 304 and the object recognition result 306 and provides the information as a position detection result 308 The position detection result. 308includes three-dimensional position information of a person or vehicle that is detected by the detection of the moving body, and three-dimensional position information of the object detection result. 306 , which is obtained through semantic segmentation.

[0038] The object tracking unit 311 performs a time series processing of the position detection result 308 based on the position detection result 308 , of the recognition result 309 the previous time and a self-propelled vehicle trajectory 310 through and gives a time series detection result 312 out. The recognition result 309 The position detection result is from the previous time. 308 until the previous time.

[0039] The object tracking unit 311 uses the recognition result 309 the previous time and the self-propelled vehicle trajectory310 to predict the position of the object detected up to the previous time at the current time. Then, a comparison is made between the position detection result and the current time. 308 A comparison is made between the current time and the predicted position obtained through position prediction. This comparison determines the difference between the position prediction result and the current time. 308 For each predicted position, a combination that minimizes the overall distance can be calculated. The distance calculation can use the proximity of the region in the image or the distance in three-dimensional space.

[0040] Then, the same ID as at the previous time is assigned to the compared object, and a new ID is assigned to the object not compared. If an object is present that was compared at the previous time, the object's speed is calculated from the position information at the previous time and the current time. The processing described above is performed on each object detected by the object recognition unit. 305 is detected, and the class, position, speed, and ID of each object are recorded as a time series detection result. 312 determined.

[0041] The map information 313 This is information obtained by converting the class information of each pixel, which is obtained through semantic segmentation, into the position detection result. 308 using the parallax image 304and an overhead view of the vehicle is obtained. The map information also includes... 313 also information contained in the time series detection result 312 are contained in the Fig. 4 shown form.

[0042] Fig. 4 is a diagram that shows a configuration example of the map information in Fig. 3 represents.

[0043] In Fig. 4 show the map information 313 multiple parts of layer information 401 up. The shift information 401 are obtained by organizing information around the vehicle for all position information. The shift information 401 This is information obtained by cutting out an area around the vehicle and subdividing that area with a grid. The information for each cell 402The cells separated by the grid correspond to the actual position information. In the case of information expressed in one-dimensional binary terms, such as road information, for example, 1 is stored in a cell corresponding to the road's position information, and 0 is stored in a cell corresponding to position information other than the road.

[0044] In the case of information expressed as a two-dimensional continuous value, such as speed information, a first directional speed component and a second directional speed component are also stored in the layer information across two layers. Here, the first direction and the second direction can represent, for example, the vehicle's direction of travel, the lateral direction, north, east, and the like. In a case where the speed information is converted into layer information, the information in the cell is also... 402 stored, which corresponds to the position information where the vehicle is located 101 or the moving body exists.

[0045] As described above, the layer information 401 Information stored in the cell 402stored that corresponds to the positional information of the captured information about a layer whose dimension is equal to or smaller than the captured information of the recognition unit 202 This includes environmental information, information about the moving body, and information about the vehicle itself. In cases where the captured information relates to information that exists only in a specific position, such as a falling object or a moving body, the information within the cell is also recorded. 402 The corresponding position information is stored. The map information 313 exhibit a structure in which different layers of information are displayed 401 , in which information about the vehicle is organized for all position information, are stacked. If the layer information 401 When stacked, the cell's position information is displayed. 402each layer was compared.

[0046] Furthermore, in the embodiment described above, the configuration in which the map information is stored was 313 generated based on the stereo camera image, as described. However, if the map information 313 To obtain the three-dimensional position, velocity, and environment of the object, object detection in the camera image and three-dimensional position detection by the lidar rider can be combined, or a configuration using other sonar devices or a configuration with only a monocular camera can be used. Map information can also be used. Furthermore, the processing performed by the stereo comparator can be enhanced. 303 , the object recognition unit 305 and the object tracking unit 311 The procedure that is carried out may be replaced by another alternative procedure.

[0047] Fig. Figure 5 is a block diagram representing a configuration of a behavior prediction unit used in the device for predicting the behavior of a moving body according to the first embodiment. This behavior prediction unit can be applied to the first behavior prediction unit. 203 or the second behavioral prediction unit 207 in Fig. 2 can be applied.

[0048] In Fig. 5 includes the behavioral prediction unit recurrent neural networks. 502-1 until 502-N , total coupled layers 505-1 until 505-N and multiplication layers 506-1 until 506-N are for each of N (N is a positive integer) moving bodies 1 up to N is provided. Furthermore, a summation layer is included in the behavioral prediction unit. 507 , folding layers 509 and 511 and a coupled layer 510together for the N moving bodies 1 up to N.

[0049] The behavioral prediction unit performs a position prediction using recurrent neural networks. 502-1 until 502-N for each of the moving bodies 1 up to N around the own vehicle 101 through. The moving bodies 1 N to N are N objects that are detected by the object recognition unit. 305 the recognition unit 202 be recognized. In the example of Fig. 1 are the moving bodies 1 up to N other vehicles 102 and 103 and the pedestrian 104 Then the convolutional neural network predicts the behavior considering that the intermediate states of the recurrent neural networks 502-1 until 502-N the moving body 1 up to N, the road conditions and traffic conditions around the vehicle are aggregated. 101can be combined and there is an interaction between the moving bodies 1 up to N and the road information are mutually influenced.

[0050] The recurrent neural networks 502-1 until 502-N These can be ordinary recurrent neural networks or derivative systems of recurrent neural networks such as a Gated Recurrent Unit (GRU) and a Long Short-Term Memory (LSTM).

[0051] Each of the recurrent neural networks 502-1 until 502-N receives movement data 501-1 until 501-N the current era of moving bodies 1 up to N and gives the movement data 503-1 until 503-N the future age of moving bodies 1 up to N. The movement data 501-1 until 501-N the current era of moving bodies 1 N to N are the magnitudes of movement of the moving bodies 1up to N since time t. This degree of movement indicates how much each of the moving bodies has changed. 1 until N has moved from before time t. The movement data 503-1 until 503-N the future age of moving bodies 1 N to N are the magnitudes of movement of the moving bodies 1 up to N at the future time. This degree of motion indicates how far each of the moving bodies will move up to the future time t0, t1, ..., tT. The motion data 501-1 until 501-N the current era of moving bodies 1 up to N and the movement data 503-1 until 503-N the future age of moving bodies 1 N to N are coordinates based on the current time position of each of the moving bodies. 1 to N.

[0052] The movement data 503-1 until 503-N the future age of moving bodies 1The numbers up to N are used to predict the direction in which the moving bodies will move. 1 They are likely to move up to N and are not accurate predictive information. Therefore, they are not used as a result of behavioral prediction.

[0053] The movement data 503-1 until 503-N the future age of moving bodies 1 up to N are used to make teaching recurrent neural networks easier. 502-1 until 502-N used. When the recurrent neural networks 502-1 until 502-N The magnitudes of motion of the moving bodies can be taught at future times t0, t1, ..., tT. 1 up to N as teacher information from the movement data 503-1 until 503-1 the future age of moving bodies 1 up to N.

[0054] The total coupled layers 505-1 until 505-Nreceive the relative position data 504-1 until 504-N the current era of moving bodies 1 up to N and output a result obtained by applying an affine transformation and an activation function. The relative position data 504-1 until 504-N the current era of moving bodies 1 numbers up to N give the relative positions of the moving bodies. 1 up to N in a coordinate system centered on the vehicle's position, at the current time. The outputs of all coupled layers. 505-1 until 505-N exhibit the same dimensions as the internal states of recurrent neural networks 502-1 until 502-N on.

[0055] The multiplication layers 506-1 until 506-N give the products of the internal states of the recurrent neural networks 502-1 until 502-Nand the expenditures of the total coupled layers 505-1 until 505-N for each element. The range of motion of each of the moving bodies. 1 up to N, which will be determined in the future by the recurrent neural networks 502-1 until 502-N The prediction is carried out in a coordinate system based on the current time of each of the moving bodies. 1 up to N. Therefore, the relative position of each of the moving bodies is determined. 1 up to N in relation to the own vehicle 101 with the value that is determined by the entire coupled layer 505-1 until 505-N The data is processed and multiplied for each element, so that the relative extent of movement to the vehicle itself is... 101 can be calculated.

[0056] The summation layer 507 calculates the summation of the outputs of the multiplication layers 506-1 until 506-N the moving body1 up to N. The summation layer 507 takes the sum of the values ​​of the multiplication layers 506-1 until 506-N from each of the moving bodies 1 up to N, so that it is possible to detect whether the moving bodies 1 to N from own vehicle 101 , to which relative position and in which direction they will move.

[0057] If the sum of the outputs of the multiplication layers 506-1 until 506-N all recognized moving bodies 1 up to N through the summation layer 507 The prediction is made taking into account the interaction between each of the moving bodies. 1 up to N and the street information were processed by the convolutional neural network. The map data 508 are data containing road information about the vehicle 101 are stored.

[0058] At this time, a folding layer is applied. 509 a convolutional neural network applied to the map data 508 on. The coupled layer 510 couples the output of the convolution layer 509 and the output of the summation layer 507 .

[0059] The output of the convolution layer 509 and the output of the summation layer 507 This can be achieved, for example, by adding the output of the summing layer. 507 with the width and height of the folding layer 509 in the channel direction of the output result of the convolution layer 509 They can be combined. Furthermore, an additional neural network, such as a convolution layer, can be placed between the summation layer. 507 and the coupled layer 510 to be added.

[0060] A folded layer 511 applies a convolutional neural network to the combined result of the summation layer output. 507and the output of the convolution layer 509 and provides behavioral data 512 of the future. The behavioral data 512 of the future represent the probability that the moving bodies 1 up to N at the coordinates at future times t0, t1, ..., tT in the coordinate system around the own vehicle 101 exist. The behavioral data 512 The future time will have the same format as the map information 313 on, which in Fig. 4 are shown.

[0061] The folding layers 509 and 511 They do not necessarily have to be a single layer and can be multiple layers, and the map data 508 , the fold layers 509 and 511 and the coupled layer 510 Each can define an intermediate state and the width and height of the output via the behavioral data. 512The number of bodies can be kept constant over time or can be reduced or increased. In the embodiment described above, the configuration was implemented in a situation where N moving bodies are present. 1 The number of moving bodies is described as existing up to N. However, the number of moving bodies is not limited, and only one or more moving bodies are required.

[0062] The processing described above determines the initial prediction behavior. 204 and a second predictive behavior 208 from the first behavioral prediction unit 203 and the second behavioral prediction unit 207 in Fig. 2 output. The first prediction behavior 204 is incorporated into the prediction error calculation unit 205 , the control unit 209 and the display unit 30 entered. The second prediction behavior 208 is integrated into the control unit 209 and the display unit 30entered.

[0063] The prediction error calculation unit 205 calculates a prediction error of the first prediction behavior 204 , which is from the first behavioral prediction unit 203 is output. Here, the initial prediction behavior is shown. 204 to the future times t0, t1, ..., tT, which are in a coordinate system around the own vehicle 101 is expressed, and a prediction error of the object position determined by the detection unit 202 The object positions detected by the recognition unit are obtained after the future times t0, t1, ..., tT. 202 Future times t0, t1, ..., tT will be recognized in the same format as the map information. 313 converted, which in Fig. 4 are shown, similar to the first prediction behavior. 204 . On the map information 313A transformation is performed such that if an object exists in a specific grid at a future time t0, t1, ..., tT, it becomes 1, and if it does not, it becomes 0. The prediction error can be determined by the mutual entropy of the initial prediction behavior. 204 and the one that is obtained by converting the recognition result into a map printout at the future times t0, t1, ..., tT.

[0064] The first parameter update magnitude calculation unit 206 can determine the extent of the update of the parameter of the first behavioral prediction unit 203 calculate to minimize the prediction error caused by the prediction error calculation unit 205 is calculated. The update rate of this parameter can be determined using a stochastic gradient descent method. The parameters of the first behavioral prediction unit 203These are weight matrices and distortion terms used in recurrent neural networks. 502-1 until 502-N , the total coupled layers 505-1 until 505-N and the fold layers 509 and 511 are included.

[0065] The control unit 209 controls the own vehicle 101 based on the initial prediction behavior 204 and the second predictive behavior 208 The control unit 209 determines the trajectory of the vehicle 101 and controls the steering angle and speed of the vehicle. 101 , in order to follow the specified trajectory. The trajectory is a set of target positions of the vehicle. 101 at certain future times t0, t1, ..., tT.

[0066] Fig. 6 is a block diagram showing a configuration of the control unit in Fig. 2 represents.

[0067] In Fig. The control unit comprises 6 209 a trajectory generation unit 601 , a trajectory evaluation unit 602 , a trajectory determination unit 603 and a trajectory tracking unit 604 .

[0068] The trajectory generation unit 601 generates multiple trajectory candidates for the own vehicle 101 The trajectory candidates can, for example, be several random trajectories.

[0069] The trajectory evaluation unit 602 evaluates multiple trajectories generated by the trajectory generation unit 601 a trajectory can be generated. A trajectory can be well evaluated if the initial predictive behavior is accurate. 204 and the second predictive behavior 208and the spatial overlap of the generated vehicle trajectory at future times t0, t1, ..., tT is small. Furthermore, the evaluation of the trajectory can be performed simultaneously with the evaluation based on the vehicle's speed and acceleration. 101 without dependence on the first prediction behavior 204 and from the second prediction behavior 208 The procedure is carried out, but includes elements for evaluating the predicted behavior of at least the moving bodies. 1 to N.

[0070] The trajectory determination unit 603 determines the trajectory with the lowest rating value of the trajectory rating unit 602 as a trajectory that the own vehicle 101 This should follow. Furthermore, the trajectory determination unit can 603 the trajectory, which is determined by the vehicle 101 to be followed, synchronously with the control cycle of the control unit 209determine.

[0071] The trajectory tracking unit 604 controls the steering angle and speed of the vehicle 101 , in order to follow the vehicle's own trajectory determined by the automatic determination unit 603 is determined.

[0072] The driving assessment unit 210 evaluates driving based on the tax result of the owner's own vehicle 101 through the control unit 209 This driving assessment determines whether the owner's own vehicle 101 Unsafe driving behavior, such as sudden braking, sudden steering, sudden acceleration, and sudden deceleration, can be determined based on whether a driver assistance function, such as a collision avoidance function of the vehicle, was activated. 101The system was activated to determine whether the steering angle and speed changed by a threshold value or more. Furthermore, this assessment makes it possible to determine whether the vehicle was... 101 has carried out an ineffective operation in which the company vehicle 101 does not move despite the fact that the moving bodies 1 up to N not about the own vehicle 101 exist and the own vehicle 101 can drive safely.

[0073] The reward generation unit 211 generates a reward based on the driving assessment result by the driving assessment unit 210 At this time, in a case where the driving assessment unit 210If it is determined that unsafe driving or ineffective driving has occurred, a negative reward may be generated, and in a case where it is determined that neither unsafe driving nor ineffective driving has occurred, a positive reward may be generated.

[0074] The second parameter update magnitude calculation unit 212 calculates an update level of the parameter of the second behavioral prediction unit 207 , so that the reward generation unit 211 The generated reward can be obtained in greater quantities. This update magnitude can be calculated using a stochastic gradient descent method or an evolutionary algorithm. At this time, the second behavioral prediction unit can be... 207 Update parameters so that unsafe driving and ineffective driving of the own vehicle 101 as a result of the actual control of the vehicle 101based on the initial prediction behavior 204 and the second predictive behavior 208 do not occur.

[0075] Since the first behavioral prediction unit 203 Taught through supervised learning, it stores the initial predictive behavior 204 strongly the frequent pattern. In a case where the control unit 209 the own vehicle 101 based on the initial prediction behavior 204 controls that strongly remember the common pattern, can control the vehicle's own vehicle 101 Drive safely when the moving bodies 1 up to N around the own vehicle 101 behave according to the frequent pattern, even if the second predictive behavior 208 nothing predicts.

[0076] In a case where the moving bodies 1 up to N around the own vehicle 101not acting according to the frequent pattern, that is, in a case where the rare pattern occurs, an unsafe event occurs and the own vehicle 101 drives unsafely when the second behavioral prediction unit 207 It predicts nothing. Since the second behavioral prediction unit 207 By teaching people to avoid such unsafe driving, she ultimately predicts the rare pattern that leads to unsafe driving.

[0077] By teaching the second behavioral prediction unit 207 , so that ineffective driving does not occur, it is also possible to prevent a situation in which the surroundings of the own vehicle 101 are dangerous and the vehicle 101 cannot be moved. At this time, the first behavioral prediction unit can be activated. 203 perform an optimistic behavior prediction and the second behavior prediction unit 207can make a cautious prediction of behavior.

[0078] Furthermore, the second behavioral prediction unit says 207 behavior that leads to unsafe driving, in the same format as the map information 313 previously, which in Fig. Figure 4 illustrates this. Therefore, it is possible that unsafe driving can be induced even in an area where the moving bodies 1 up to N not about the own vehicle 101 exist, even an area in which the moving bodies 1 until N can suddenly appear due to jumping out, such as at an intersection, is not affected, and it is possible to observe the behavior of the appearance of the moving bodies. 1 to predict up to N.

[0079] Furthermore, the reward generation unit can 211 the reward synchronized with the control cycle of the control unit 209Updating can update the reward for each segment of the route or combine them. A segment of the route could be, for example, a left turn, a right turn, a straight line to an intersection, or a departure point for a destination on a map used for navigation. In a case where the control unit's control cycle 209 and the route segments can be combined; these can be treated equally, or any one of them can be weighted. The first behavioral prediction unit 203 and the second behavioral prediction unit 207 can the first predictive behavior 204 and the second predictive behavior 208 synchronous with the reward update period of the reward generation unit 211 update.

[0080] Fig. 7(a) is a schematic diagram illustrating an evaluation procedure of the driving assessment unit of Fig. 2 represents, Fig. 7(b) is a diagram that shows an example of the data map in Fig. 5 represents, and Fig. 7(c) is a diagram that shows an example of future time behavioral data from Fig. 5 represents.

[0081] In Fig. 7(a) it is assumed that the own vehicle 101 on the street 100 drives and the other vehicle 105 in front of the own vehicle 101 is driving. It is assumed that the other vehicle 105 along the route K5 moved. The other vehicle 105 corresponds to the moving body 1 in Fig. 5.

[0082] The street 100 is by the recognition unit 202 recognized, which is in the owner's own vehicle 101 is planned, and map data 508 are created. It is assumed that there is 1 in each cell of the map data. 508 The location of the street is stored. 100in Fig. 7(a) corresponds, and 0 corresponds to a position other than the road 100 is saved.

[0083] The movement data 501-1 the current time of the moving body 1 , the relative position data 504-1 the current time of the moving body 1 and the map data 508 of the other vehicle 105 are integrated into the behavioral prediction unit in Fig. 5 entered. As output of this behavioral prediction unit, as in Fig. As shown in 7(c), behavioral data will then be displayed. 512-0 , 512-1 , ..., 512-T of the future time at future times t0, t1, ..., tT. Each cell of the behavioral data 512-0 , 512-1 , ... 512-T of the future time stores the probability that the other vehicle 105 exists in every coordinate at future times t0, t1, ..., tT.

[0084] The control unit 209 from Fig. 2 controls the own vehicle 101 based on behavioral data 512-0 , 512-1 , ..., 512-T the future time of the other vehicle 105 . Here, it is assumed that the trajectory generation unit 601 Trajectory candidates K1-1 , K1-2 and K1-3 of the owner's vehicle 101 has generated. Then the trajectory evaluation unit evaluates 602 the spatial overlap of each of the trajectory candidates K1-1 , K1-2 and K1-3 with the other vehicle 105 at future times t0, t1, ..., tT. At this time, for example, the trajectory candidate K1-1 The spatial overlap is 0% for the trajectory candidate. K1-2 The spatial overlap is 80% and for the trajectory candidate K1-3 The spatial overlap is 30%. In this case, the trajectory determination unit is used.603 the trajectory candidates K1-1 with the smallest spatial overlap as trajectory, to which the own vehicle 101 This should follow. Then the trajectory tracking unit controls it. 604 the steering angle and speed of the vehicle 101 , to the trajectory candidate K1-1 to follow, which was determined as the vehicle's own trajectory.

[0085] It is assumed that as a result of controlling the steering angle and speed of the vehicle itself 101 , to the trajectory candidate K1-1 to follow, sudden braking and sudden steering of one's own vehicle 101 have occurred. At this time, the driving assessment unit determines 210 , that driving is unsafe, and the reward generation unit 211 This generates a negative reward. Here, the second parameter update magnitude calculation unit is calculated. 212the update level of the parameter of the second behavioral prediction unit 207 , so that more rewards are generated by the reward generation unit 211 They can be generated and obtained. Therefore, the second parameter update extent calculation unit calculates 212 the update level of the parameter of the second behavioral prediction unit 207 , so that a negative reward is not generated. Consequently, the second behavioral prediction unit 207 the second predictive behavior 208 so that the driving assessment unit 210 It does not state that driving is unsafe.

[0086] Fig. Figure 8 is a diagram showing an example of a first prediction behavior and a second prediction behavior displayed by the device for predicting the behavior of a moving body. Fig. 2 can be predicted. In Fig. 8 will be the first predictive behaviors 204-1 until 204-3 and a second predictive behavior 208-1 on a windshield 40 of the owner's vehicle 101 projected. The first predictive behavior 204-1 until 204-3 and the second predictive behavior 208-1 can be displayed in positions of the moving body that are actually seen by the driver through the windshield 40 can be observed.

[0087] This allows the driver to make initial predictions. 204-1 until 204-3 and the second predictive behavior 208-1 detects without distracting the driver from the front while driving.

[0088] In the first embodiment described above, the configuration in which the first prediction behavior 204 and the second predictive behavior 208 both through the control unit 209They are used, as described.

[0089] The following is a procedure for selecting the predicted behavior by the control unit. 209 It is used as described in the surrounding environment. (Second embodiment)

[0090] Fig. Figure 9 is a block diagram illustrating a configuration of the device for predicting the behavior of a moving body according to the second embodiment. In the device for predicting the behavior of a moving body... Fig. 9 is a prediction procedure determination unit 801 for the device for predicting the behavior of a moving body of Fig. 2 added. The prediction method determination unit 801 includes a weight estimation unit 802 .

[0091] The prediction method determination unit 801determines the predicted behavior that is determined by the control unit 209 is used according to the information from the surrounding environment, which is gathered by the detection unit. 202 to be captured as any of the first predictive behavior 204 , only the second predictive behavior 208 and a weighted average of the first prediction behavior 204 and the second predictive behavior 208 In a case where the weighted average of the first prediction behavior 204 and the second predictive behavior 208 The selected unit also estimates the weight estimation unit. 802 the weight used for the weighted average.

[0092] The prediction method is determined through supervised learning. The prediction method determination unit 801 stores the vehicle's own trajectory, which is determined by the control unit 209using only the first prediction behavior 204 is generated, and the vehicle's own trajectory, which is determined by the control unit 209 using only the second prediction behavior 208 is generated simultaneously in connection with the information from the recognition unit. 202 . The driving assessment unit will then determine the future time. 210 , whether both the self-propelled vehicle trajectory is based solely on the first prediction behavior 204 as well as the vehicle trajectory based solely on the second prediction behavior 208 do not cause unsafe or ineffective driving.

[0093] The prediction method determination unit 801 features a machine learning-based prediction model that produces two outputs on whether the self-propelled vehicle trajectory is based solely on the first prediction behavior 204 with the information from the recognition unit 202as input causes unsafe driving and ineffective driving, and whether the vehicle trajectory is based solely on the second prediction behavior 208 The predictive model is trained as a two-class classification problem: a case where the self-driving trajectory, based solely on the predicted behavior, causes unsafe and ineffective driving (a negative example) and a case where it does not cause unsafe and ineffective driving (a positive example).

[0094] At the time of actual driving, the prediction procedure determination unit uses 801 the information provided by the recognition unit 202 to be recorded in order to predict whether the self-propelled vehicle trajectory will be determined using only the first prediction behavior 204 and the self-propelled vehicle trajectory using only the second prediction behavior208 unsafe and ineffective driving is caused, and outputs a certainty factor that is a positive example. The certainty factor is that the vehicle's own trajectory is determined using only the initial prediction behavior. 204 does not cause unsafe or ineffective driving, is P1 and the certainty factor that the vehicle trajectory using only the second prediction behavior 208 Unsafe driving and inactive driving are caused by P2.

[0095] If the certainty factor P1 is greater than a threshold TH and the certainty factor P2 The predictive procedure determination unit is determined when the value is smaller than a threshold value TL. 801 that the control unit 209 only the first prediction behavior 204 used. If the certainty factor P1 is smaller than the threshold TL and the certainty factor P2The predictive procedure determination unit is determined if the value is greater than the threshold TH. 801 that the control unit 209 only the second prediction behavior 208 used.

[0096] In other cases, the initial predictive behavior 204 and the second predictive behavior 208 weighted with a ratio of P1 / (P1 + P2):P2 / (P1 + P2) and the value obtained by taking the weighted average is determined by the control unit 209 used. The threshold values ​​TH and TL are values ​​that are determined in advance. At this time, the detection unit can 202 in addition to the in Fig. Add the following information to the displayed data: GPS information, map information, and the road type of the route.

[0097] By selecting the predicted behavior that is determined by the control unit 209The first prediction behavior can be determined based on the surrounding environment. 204 and the second predictive behavior 208 Based on the certainty factor, it can be predicted that the vehicle's own trajectory will not cause unsafe or ineffective driving. The prediction accuracy of the first prediction behavior 204 and the second predictive behavior 208 can be improved. (Third embodiment)

[0098] Fig. Figure 10 is a block diagram representing a hardware configuration of a device for predicting the behavior of a moving body according to a third embodiment.

[0099] In Fig. 10 includes the device 10 A processor is used to predict the behavior of a moving body. 11 , a communication control device 12 , a communication interface 13, a main storage device 14 and an external storage device 15 The processor 11 , the communication control device 12 , the communication interface 13 , the main storage device 14 and the external storage device 15 are via an internal bus 16 interconnected. The main storage device 14 and the external storage device 15 are from the processor 11 accessible.

[0100] Furthermore, the sensor 20 , the display unit 30 and an operating unit 40 as the input / output interface of the device 10 Designed to predict the behavior of a moving body. The sensor 20 , the display unit 30 and the operating unit 40 are connected to the internal bus 16 connected. The operating unit 40This controls acceleration, deceleration, braking, steering, and the like of the vehicle. 101 by operating the engine, transmission, brakes, steering, and the like of the vehicle 101 based on a command from the control unit 209 in Fig. 2 through.

[0101] The processor 11 is hardware that enables the operation of the entire device 10 to predict the behavior of a moving body. The main storage device 14 It can be configured, for example, by a semiconductor memory such as SRAM or DRAM. The main memory device 14 can save a program that is processed by the processor 11 is executed, or a work area for the processor 11 Make it available to run the program.

[0102] The external storage device 15is a storage device with a large storage capacity, for example, a hard disk drive or an SSD (semiconductor drive). The external storage device 15 It can hold executable files from various programs. The external storage device 15 can a program 15A to store data for predicting the behavior of a moving body. The processor 11 The program reads 15A to predict the behavior of a moving body into the main memory device 14 and runs the program 15A to predict the behavior of a moving body, thereby enabling the functions of the device 10 to predict the behavior of a moving body in Fig. 1 can be realized.

[0103] The communication control device 12It is hardware with a function for controlling communication with the outside world. The communication control device 12 is connected to a network 19 via the communication interface 13 tied together.

[0104] As described above, the embodiments of the invention have been described. However, the mounting location of each function described in this embodiment is irrelevant. In other words, it can be mounted on a vehicle or on a data center that can communicate with the vehicle.

[0105] Furthermore, the embodiment described above includes a case in which the device for predicting the behavior of a moving body is used to operate a vehicle. However, the device for predicting the behavior of a moving body can also be used for objects other than vehicles, such as flying objects like drones and unmanned vehicles. It can be used for flight control or for walking and posture control of an artificial intelligence-equipped robot.

[0106] Furthermore, the invention is not limited to the embodiments described above, but may include various modifications. The embodiments described above have been described in detail for a clear understanding of the invention and are not necessarily limited to those with all the described configurations. Moreover, some of the configurations of a particular embodiment may be replaced by the configurations of other embodiments, and the configurations of other embodiments may be added to the configurations of a particular embodiment. Additionally, some of the configurations of each embodiment may be omitted, replaced by other configurations, and added to other configurations. Reference symbol list 10 Device for predicting the behavior of a moving body 20 Sensor 101 Own vehicle 102, 103 other vehicles 104 pedestrians QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2013196601 A

[0005]

Claims

[1] Device for predicting the behavior of a moving body, comprising: a first behavior prediction unit that outputs a first predicted behavior of a moving body based on a prediction result of a behavior of the moving body that is detectable by a vehicle, and a detection result of a behavior of the moving body after a prediction time has elapsed; and a second behavior prediction unit that outputs a second predictive behavior of the moving body, detectable by the vehicle, based on the vehicle's behavior. [2] Device for predicting the behavior of a moving body according to claim 1, wherein the first behavioral prediction unit comprises a first neural network that outputs the first predictive behavior based on supervised learning, and wherein the second behavioral prediction unit comprises a second neural network that outputs the second predictive behavior based on reinforcement learning. [3] Device for predicting the behavior of a moving body according to claim 1, where the first prediction behavior and the second prediction behavior are used for the vehicle's driving control, wherein the first behavioral prediction unit learns the first prediction behavior in order to minimize an error between the prediction result of the behavior of the moving body and the detection result of the behavior of the moving body after the prediction time has elapsed, and wherein the second behavior prediction unit learns the second predictive behavior of the moving body that is detectable by the vehicle, so that the vehicle does not drive unsafely. [4] Device for predicting the behavior of a moving body according to claim 3, further comprising: a recognition unit that detects the type, position, and speed of the moving body; a control unit that controls the vehicle based on the first prediction behavior and / or the second prediction behavior; a driving assessment unit that evaluates the vehicle's driving safety based on a control result of the vehicle; and a reward generation unit that generates a negative reward if a safety assessment result is uncertain, and generates a positive reward if the safety assessment result is certain. where a prediction parameter of the second behavior prediction unit is updated to maximize the reward. [5] Device for predicting the behavior of a moving body according to claim 4, wherein the detection unit comprises: a stereo comparison unit that generates a parallax image based on multiple camera images, an object recognition unit that recognizes an object based on the camera image, a position calculation unit that calculates a position detection result at a current time of the object based on the parallax image and the detection result of the object, and an object tracking unit that predicts a position detection result of the object at a current time based on a trajectory of the vehicle and the position detection result up to a previous time, and tracks the object based on a comparison result between a predicted position detection result and a position detection result calculated by the position calculation unit. [6] Device for predicting the behavior of a moving body according to claim 4, wherein the first behavior prediction unit comprises: N recurrent neural networks that, based on motion data from a current time of N (N is a positive integer) moving bodies, each output motion data for a future time of the N moving bodies, N coupled layers in total, each applying an affine transformation and an activation function to relative position data of the N moving bodies based on the position of the vehicle, N multiplication layers, each multiplying the internal states of the N recurrent neural networks and the outputs of the N coupled layers in total, a summation layer that sums the outputs of the N multiplication layers, a first convolutional layer that applies a convolutional neural network to road information around the vehicle, a coupled layer that couples an output of the summation layer and an output of the first convolution layer, and a second convolution layer that applies a convolutional neural network to an output of the coupled layer. [7] Device for predicting the behavior of a moving body according to claim 4, wherein the control unit comprises: a trajectory generation unit that generates multiple trajectory candidates for the vehicle, a trajectory evaluation unit that evaluates the trajectory candidate based on the first prediction behavior and the second prediction behavior, a trajectory determination unit that determines a trajectory of the vehicle based on an evaluation result by the trajectory evaluation unit, and a trajectory tracking unit that controls the vehicle so that the vehicle follows a trajectory determined by the trajectory determination unit. [8] Device for predicting the behavior of a moving body according to claim 4, further comprising: a prediction procedure determination unit which, based on a certainty factor as to whether the vehicle causes unsafe driving or ineffective driving, determines a predicted behavior used in the control unit as any of the first prediction behavior only, the second prediction behavior only, and a weighted mean of the first prediction behavior and the second prediction behavior. [9] Device for predicting the behavior of a moving body according to claim 4, further comprising: a weight estimation unit that estimates a weight for the first predictive behavior and a weight for the second predictive behavior, the control unit controls the vehicle based on a weighted average of the first prediction behavior and the second prediction behavior. [10] Device for predicting the behavior of a moving body according to claim 4, further comprising: a display unit that shows the first prediction behavior and the second prediction behavior together with a camera image in front of the vehicle in a superimposed manner. [11] Device for predicting the behavior of a moving body according to claim 4, further comprising: a display unit that shows the first prediction behavior and the second prediction behavior on a windshield of the vehicle. [12] Method for predicting the behavior of a moving body, comprising the following: Predictions of the initial behavior of a moving body based on supervised learning; and Predictions of a second behavior of the moving body based on reinforcement learning, where the frequency of occurrence of the second behavior at the time of prediction is lower than the frequency of occurrence of the first behavior.