Prediction Tracking Device, Prediction Tracking Method, and Computer-Readable Recording Medium

Abstract

The moving object detection unit (113) extracts point clouds representing moving objects existing around the moving body from the point cloud data obtained by the LiDAR that measures the periphery of the moving body. The moving object tracking unit (116) extracts point clouds within the range of the posterior distribution for the tracking object that is the moving object being tracked from the point cloud data, and matches the extracted point clouds with the point clouds of the moving objects.

Description

Technical Field

[0001] The present invention relates to predictive tracking of moving objects. Background Art

[0002] There is known a method of tracking a moving object using point clouds measured by LiDAR.

[0003] LiDAR is an abbreviation for Light Detection And Ranging.

[0004] In the method disclosed in Patent Document 1, the point clouds of surrounding objects are matched by the ICP algorithm to estimate the own position. In addition, a prediction filter such as a Kalman filter or Bayesian estimation is used to estimate the moving position of surrounding vehicles.

[0005] ICP is an abbreviation for Iterative Closest Point.

[0006] Prior Art Documents

[0007] Patent Documents

[0008] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2018-141716 Summary of the Invention

[0009] Problems to be Solved by the Invention

[0010] In the prior art, a prediction filter is used to predict the movement of an object to be tracked, and the posterior probability is maximized. As a result, the estimation accuracy is improved.

[0011] However, estimation cannot be performed without a prior distribution, and the object to which the prior distribution is given is limited to a vehicle.

[0012] In addition, in the maximization of the posterior probability, the posterior probability may fall into a local solution. In the case of an object other than a vehicle, the value does not converge according to the assignment of the prior distribution, and prediction becomes difficult.

[0013] In addition, since the ICP algorithm is used for a large range or a plurality of objects, the calculation cost increases.

[0014] An object of the present invention is to improve the predictive tracking accuracy of a moving object while suppressing the calculation cost.

[0015] Means for Solving the Problems

[0016] The predictive tracking device of the present invention includes:

[0017] A moving object detection unit that extracts point clouds representing moving objects existing around the moving body from the point cloud data obtained by a LiDAR that measures the periphery of the moving body; and

[0018] A moving object tracking unit that extracts point clouds within the range of the posterior distribution of the tracking object that is the moving object being tracked from the point cloud data, and performs matching between the extracted point clouds and the point clouds of the moving objects.

[0019] Advantages of the Invention

[0020] According to the present invention, it is possible to provide prediction tracking accuracy of a moving object while suppressing the computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a structural diagram of the prediction tracking system 200 in Embodiment 1.

[0022] Figure 2 is a structural diagram of the prediction tracking device 100 in Embodiment 1.

[0023] Figure 3 is a structural diagram of the sensor group 180 in Embodiment 1.

[0024] Figure 4 is a flowchart of the prediction tracking method in Embodiment 1.

[0025] Figure 5 is a flowchart of the prediction tracking method in Embodiment 1.

[0026] Figure 6 is a flowchart of the prediction tracking method in Embodiment 1.

[0027] Figure 7 is a flowchart of the prediction tracking method in Embodiment 1.

[0028] Figure 8 is an explanatory diagram related to the prediction filter in Embodiment 1.

[0029] Figure 9 is an explanatory diagram related to the prediction tracking in Embodiment 1.

[0030] Figure 10 is an explanatory diagram related to the prediction tracking in Embodiment 1.

[0031] Figure 11 is an explanatory diagram related to the prior distribution and likelihood function of the category of the moving object in Embodiment 1.

[0032] Figure 12It is an explanatory diagram related to the prior distribution and likelihood function of the moving direction of a moving object in Embodiment 1.

[0033] Figure 13 It is a diagram showing an example of the sensor group 180 in Embodiment 1.

[0034] Figure 14 It is a structural diagram of the prediction tracking device 100 in Embodiment 2.

[0035] Figure 15 It is a hardware structural diagram of the prediction tracking device 100 in the embodiment. Detailed Embodiment

[0036] In the embodiment and the drawings, the same reference numerals are assigned to the same elements or corresponding elements. The description of the elements with the same reference numerals as those already described is appropriately omitted or simplified. The arrows in the drawings mainly show the data flow or processing flow.

[0037] Embodiment 1

[0038] According to Figures 1 to 13 , the prediction tracking system 200 will be described.

[0039] ***Description of the Structure***

[0040] According to Figure 1 , the structure of the prediction tracking system 200 will be described.

[0041] The prediction tracking system 200 is mounted on the vehicle 210.

[0042] The vehicle 210 is an automobile on which the prediction tracking system 200 is mounted. An automobile is an example of a moving body.

[0043] The prediction tracking system 200 includes a prediction tracking device 100 and a sensor group 180.

[0044] According to Figure 2 , the structure of the prediction tracking device 100 will be described.

[0045] The prediction tracking device 100 is a computer having hardware such as a processor 101, a memory 102, an auxiliary storage device 103, a communication device 104, and an input / output interface 105. These hardware components are interconnected via signal lines.

[0046] The processor 101 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 101 is a CPU, a DSP, or a GPU.

[0047] IC is an abbreviation for Integrated Circuit.

[0048] The CPU is short for Central Processing Unit.

[0049] The DSP is short for Digital Signal Processor.

[0050] The GPU is short for Graphics Processing Unit.

[0051] The memory 102 is a volatile or non-volatile storage device. The memory 102 is also referred to as the main storage device or main memory. For example, the memory 102 is RAM. The data stored in the memory 102 is saved to the auxiliary storage device 103 as needed.

[0052] RAM is short for Random Access Memory.

[0053] The auxiliary storage device 103 is a non-volatile storage device. For example, the auxiliary storage device 103 is a ROM, HDD, or flash memory. The data stored in the auxiliary storage device 103 is loaded into the memory 102 as needed.

[0054] ROM is short for Read Only Memory.

[0055] HDD is short for Hard Disk Drive.

[0056] The communication device 104 is a receiver and a transmitter. For example, the communication device 104 is a communication chip or NIC.

[0057] NIC is short for Network Interface Card.

[0058] The input / output interface 105 is a port connected to an input device, an output device, and the sensor group 180. For example, the input / output interface 105 is a USB terminal, the input device is a keyboard and a mouse, and the output device is a display.

[0059] USB is short for Universal Serial Bus.

[0060] The prediction tracking device 100 has elements such as a sensor data acquisition unit 111, a self-position estimation unit 112, a moving object detection unit 113, a moving object recognition unit 114, a movement prediction unit 115, and a moving object tracking unit 116. These elements are implemented by software.

[0061] In the auxiliary storage device 103, a prediction tracking program is stored for causing a computer to function as a sensor data acquisition unit 111, a self-position estimation unit 112, a moving object detection unit 113, a moving object recognition unit 114, a movement prediction unit 115, and a moving object tracking unit 116. The prediction tracking program is loaded into the memory 102 and executed by the processor 101.

[0062] An OS is also stored in the auxiliary storage device 103. At least a part of the OS is loaded into the memory 102 and executed by the processor 101.

[0063] The processor 101 executes the prediction tracking program while executing the OS.

[0064] The OS is an abbreviation for Operating System.

[0065] The input / output data of the prediction tracking program is stored in the storage unit 190.

[0066] The memory 102 functions as the storage unit 190. However, storage devices such as the auxiliary storage device 103, registers in the processor 101, and flash memory in the processor 101 can also function as the storage unit 190 instead of or together with the memory 102.

[0067] The prediction tracking device 100 may also have a plurality of processors that replace the processor 101. The plurality of processors share the functions of the processor 101.

[0068] The prediction tracking program can be recorded (saved) in a non-volatile recording medium such as an optical disc or flash memory in a computer-readable manner.

[0069] According to Figure 3 , the structure of the sensor group 180 is described.

[0070] The sensor group 180 includes sensors such as a LiDAR 181, a GPS 182, and a speedometer 183.

[0071] As an example of the LiDAR 181, a laser scanner is provided. The LiDAR 181 emits laser light in various directions, receives the laser light reflected from each location, and outputs point cloud data. The point cloud data represents a distance vector and a reflection intensity for each location where the laser light is reflected. LiDAR is an abbreviation for Light Detection and Ranging.

[0072] GPS 182 is an example of a positioning system. GPS 182 receives positioning signals, locates its own position, and outputs positioning data. The positioning data represents position information. The position information represents three-dimensional coordinate values. GPS is short for Global Positioning System.

[0073] The speedometer 183 measures the speed of the vehicle 210 and outputs speed data. The speed data represents the speed of the vehicle 210.

[0074] ***Description of the operation***

[0075] The operation process of the prediction tracking device 100 is equivalent to the prediction tracking method. In addition, the operation process of the prediction tracking device 100 is equivalent to the processing process based on the prediction tracking program.

[0076] According to Figures 4 to 7 , the prediction tracking method will be described.

[0077] In step S101, the sensor data acquisition unit 111 acquires a set of sensor data from the sensor group 180.

[0078] The sensor group 180 is a collection of sensors. The set of sensor data is a collection of sensor data. Sensor data is data obtained by sensors.

[0079] The set of sensor data includes sensor data such as point cloud data, positioning data, and speed data.

[0080] In step S102, the own position estimation unit 112 uses the set of sensor data to estimate the position of the vehicle 210. The position of the vehicle 210 is referred to as the "own position".

[0081] For example, the own position estimation unit 112 uses the speed indicated by the speed data and the elapsed time since measurement to calculate the movement amount. Then, the own position estimation unit 112 estimates the own position based on the position information indicated by the positioning data and the calculated movement amount.

[0082] For example, the own position estimation unit 112 uses the point cloud data to match the point cloud of the ground objects through the SLAM technology and estimates the own position.

[0083] In step S103, the moving object detection unit 113 extracts the point cloud indicating the moving objects existing around the vehicle 210 from the point cloud indicated by the point cloud data.

[0084] A moving object is an object that moves such as a car. That is, both a moving car and a stationary car are equivalent to moving objects. However, the moving object is not limited to a vehicle.

[0085] Specifically, the moving object detection unit 113 extracts the point cloud of the moving object through machine learning, rule-based methods, or deep learning.

[0086] In the rule-based method, first, the point cloud of the road surface is detected, and the point cloud of the moving object is extracted from the point cloud other than the point cloud of the road surface.

[0087] The moving object corresponding to the extracted point cloud is called a "detected object".

[0088] In step S104, the moving object detection unit 113 calculates the relative position of the detected object based on the point cloud of the moving object.

[0089] The relative position of the detected object is the position of the detected object relative to the own vehicle 210.

[0090] Specifically, the moving object detection unit 113 selects one representative point from the point cloud of the moving object and converts the distance vector of the representative point into three-dimensional coordinate values. The three-dimensional coordinate values obtained by the conversion represent the relative position of the detected object.

[0091] In step S105, the moving object recognition unit 114 determines the attributes of the detected object based on the point cloud of the moving object.

[0092] The attributes of the detected object include category, direction, and size, etc. The category distinguishes the types of moving objects such as trucks, ordinary vehicles, motorcycles, bicycles, or pedestrians. The direction refers to the heading of the moving object, which is equivalent to the moving direction. The size represents width, depth, height, etc.

[0093] Specifically, the moving object recognition unit 114 determines the attributes of the moving object through machine learning, rule-based methods, or deep learning.

[0094] In step S106, the moving object tracking unit 116 extracts the point cloud within the range of the posterior distribution of the tracking object from the point cloud data.

[0095] The tracking object is the moving object being tracked, that is, the moving object that the prediction tracking device 100 is tracking.

[0096] Regarding the posterior distribution, it will be described later.

[0097] The moving object tracking unit 116 performs matching between the extracted point cloud and the point cloud of the detected object.

[0098] For example, the moving object tracking unit 116 performs matching through the ICP algorithm. ICP is the abbreviation of Iterative Closest Point.

[0099] In the case where there are a plurality of point clouds within the posterior distribution, the moving object tracking unit 116 performs matching in the order from the largest to the smallest likelihood based on the posterior distribution.

[0100] In step S107, the moving object tracking unit 116 determines whether there is a point cloud that matches the point cloud of the detected object among the point clouds within the posterior distribution of the tracking object based on the matching result.

[0101] In the case where there is a point cloud that matches the point cloud of the detected object, the process proceeds to step S111.

[0102] In the case where there is no point cloud that matches the point cloud of the detected object, the process proceeds to step S121.

[0103] In step S111, the moving object tracking unit 116 uses the matching result to modify the relative position of the detected object calculated in step S104. That is, the moving object tracking unit 116 calculates the accurate relative position of the detected object.

[0104] The accurate relative position is represented by a group of relative distances (x, y, z) and rotation angles (θx, θy, θz).

[0105] In step S112, the movement prediction unit 115 updates the parameters of the prediction filter for the tracking object based on the relative position of the detected object.

[0106] A specific example of the prediction filter is a Kalman filter. The parameters of the Kalman filter are the predicted value, the covariance of the state transition matrix, the observation noise, and the Kalman gain.

[0107] In the prediction step of the Kalman filter, the predicted value and the covariance of the state transition matrix are updated based on the covariance of the system noise and the Jacobian matrix of the state transition.

[0108] In the measurement update step of the Kalman filter, the Kalman gain is updated using the covariance of the state transition matrix and the observation noise.

[0109] In a general Kalman filter, the Kalman gain is updated as follows. The relative position obtained in step S111 is referred to as the accurate relative position.

[0110] In the previous step S112, the movement prediction unit 115 predicts the current relative position of the detected object by operating the Kalman filter. The predicted relative position (predicted value) is referred to as the predicted relative position.

[0111] In the current step S112, the movement prediction unit 115 calculates the difference between the accurate relative position and the predicted relative position. Then, the movement prediction unit 115 uses the calculated difference to update the Kalman gain.

[0112] In the case where Bayesian estimation is used as the prediction filter, the parameters of the prediction filter are the predicted values, hyperparameters, etc. The hyperparameters are used to determine the distribution shape of the likelihood (probability distribution) of the observation noise. The likelihood (probability distribution) of the observation noise is updated by the EM algorithm or an approximation method.

[0113] In step S113, the movement prediction unit 115 calculates the posterior distribution for tracking the object using the prediction filter for tracking the object, the prior distribution, and the likelihood function.

[0114] Then, the movement prediction unit 115 predicts the state quantity of the tracked object using the posterior distribution for tracking the object.

[0115] The state quantity to be predicted is the state quantity with the maximum posterior probability. The state quantity with the maximum posterior probability is calculated by maximum a posteriori probability estimation.

[0116] For example, the state quantity is position, velocity, acceleration, yaw angle, yaw rate, size, etc. In addition, the prior distribution, likelihood function, and posterior distribution are set according to the state quantity.

[0117] In step S114, the moving object tracking unit 116 determines whether there is a tracking ID for detecting the object.

[0118] In the case where there is a tracking ID for the detected object, the tracking ID for the detected object is continued to be used. The process proceeds to step S116.

[0119] In the case where there is no tracking ID for the detected object, the process proceeds to step S115.

[0120] In step S115, the moving object tracking unit 116 generates a new tracking ID for detecting the object and attaches the new tracking ID to the point cloud of the detected object.

[0121] In step S116, the moving object tracking unit 116 adds the sampling time to the tracking duration for the detected object.

[0122] The tracking duration is the length of time for which the object being tracked is being tracked. The initial value of the tracking duration is zero seconds.

[0123] The sampling time is, for example, the time interval for acquiring the sensor data group in step S101.

[0124] In step S117, the moving object tracking unit 116 sets the posterior distribution for tracking the object as the new prior distribution for tracking the object. Thus, the posterior distribution for tracking the object is used as the prior distribution for tracking the object in the processing after the next step S101.

[0125] After step S117, the process proceeds to step S101.

[0126] In step S121, the moving object tracking unit 116 determines whether there is at least one tracking ID.

[0127] In the case of having a tracking ID, the existing tracking ID is continued to be used. The process proceeds to step S131.

[0128] In the case of not having a tracking ID, the process proceeds to step S122.

[0129] In step S122, the moving object tracking unit 116 generates a new tracking ID for detecting an object and attaches the new tracking ID to the point cloud of the detected object. Thus, the detected object is treated as a new tracking object.

[0130] In step S123, the moving object tracking unit 116 makes an initial setting of the prior distribution, likelihood function, and prediction filter according to the attributes of the detected object for detecting the object.

[0131] In step S124, the moving object tracking unit 116 uses the prior distribution, likelihood function, and prediction filter for the detected object to predict the state quantity of the detected object.

[0132] The prediction method is the same as the method in step S113.

[0133] In step S125, the moving object tracking unit 116 sets the sampling time to the tracking duration for the detected object.

[0134] In step S126, the moving object tracking unit 116 sets the posterior distribution for the detected object to the new prior distribution for the detected object. Thus, the detected object is treated as a tracking object, and the posterior distribution for the detected object is used as the prior distribution for the detected object (tracking object) in the subsequent processing after the next step S101.

[0135] After step S126, the process proceeds to step S101.

[0136] The steps after step S131 are described. In the description of the steps after step S131, the "tracking object" refers to the tracking object identified by the tracking ID found in step S121.

[0137] In step S131, the movement prediction unit 115 updates the parameters of the prediction filter for the tracking object according to the previous predicted state quantity of the tracking object.

[0138] In step S132, the movement prediction unit 115 uses the prior distribution, likelihood function, and prediction filter for the tracking object to predict the state quantity of the tracking object.

[0139] The prediction method is the same as the method in step S113.

[0140] In step S133, the moving object tracking unit 116 determines whether the tracking interruption time for tracking the object is less than the tracking suspension time.

[0141] The tracking interruption time is the time length of the tracking interruption for the tracked object, which is equivalent to the time length of the point cloud where the tracked object is not detected.

[0142] The tracking suspension time is the upper limit of the tracking interruption time and is predefined.

[0143] When the tracking interruption time for tracking the object is less than the tracking suspension time, the process proceeds to step S134.

[0144] When the tracking interruption time for tracking the object is equal to or greater than the tracking suspension time, the process proceeds to step S135.

[0145] In step S134, the moving object tracking unit 116 adds the sampling time to the tracking interruption time for tracking the object.

[0146] After step S134, the process proceeds to step S136.

[0147] In step S135, the moving object tracking unit 116 discards the tracking ID for the tracked object. Thus, the tracking of the tracked object is aborted.

[0148] After step S135, the process proceeds to step S136.

[0149] In step S136, the moving object tracking unit 116 sets the posterior distribution for the tracked object as the new prior distribution for the tracked object. Thus, the posterior distribution for the tracked object is used as the prior distribution for the tracked object in the subsequent processing after the next step S101.

[0150] After step S136, the process proceeds to step S101.

[0151] According to Figure 8 , a Bayesian estimation, which is an example of a prediction filter, will be described.

[0152] The movement prediction unit 115 predicts the future position of the moving object using, for example, Bayesian estimation.

[0153] Figure 8 A simple example of Bayesian estimation is shown. The horizontal axis represents the position of a vehicle, which is an example of a moving object. The vertical axis represents the probability density that the vehicle exists at each position.

[0154] The prior distribution P(A) represents the probability of event A occurring. Event A is equivalent to the state quantity of a moving object. The prior distribution is also called the prior probability distribution.

[0155] The posterior distribution P(A|X) represents the probability of event A occurring under the condition that event X occurs when the likelihood function is P(X|A) (conditional probability). Event X is equivalent to the state quantity of a moving object. The posterior distribution is also called the posterior probability distribution.

[0156] As Figure 8 shown, the posterior distribution P(A|X) forms a complex shape with respect to the prior distribution P(A) and the likelihood function P(X|A).

[0157] In order to find the vehicle position with the maximum posterior probability (optimal solution), it is necessary to solve the optimal solution analytically or numerically.

[0158] The following formula represents Bayesian estimation. Σ with the subscript "A" refers to the sum of values for all events A.

[0159] The movement prediction unit 115 calculates the following formula to calculate the predicted value  that maximizes the posterior distribution P(A|X) and the posterior distribution P(A|X). In addition, the initial values of the prior distribution P(A) and the likelihood function P(X|A) vary according to the attributes of the moving object.

[0160] [Equation 1]

[0161]

[0162] The optimal solution with the maximum posterior probability needs to be obtained analytically or numerically. Obtaining the optimal solution with the maximum posterior probability is called maximizing the posterior probability.

[0163] When obtaining the optimal solution analytically, the prior distribution is a conjugate prior distribution, and a posterior distribution with the same distribution as the prior distribution can be derived. When the prior distribution is a conjugate prior distribution, the optimal solution can be obtained analytically. In this case, approximate estimation methods such as the variational method or Laplace approximation can be used to obtain an approximation of the posterior probability.

[0164] When obtaining the optimal solution numerically, it is necessary to solve an optimization problem. To solve the optimization problem, there are the EM algorithm or the MCMC method, etc.

[0165] In the EM algorithm, the derivative of the posterior distribution is not required. EM is the abbreviation of Expectation - Maximization (expectation maximization).

[0166] The MCMC method is a sampling method that approximates a distribution. MCMC is short for Markov Chain Monte Carlo Method.

[0167] According to Figure 9 and Figure 10 , the operations of the movement prediction unit 115 and the moving object tracking unit 116 are described. The solid circle represents the prior distribution, the dashed circle represents the likelihood function, and the dotted line circle represents the posterior distribution.

[0168] Consider the position V of the host vehicle at time t t (Refer to Figure 9 ) as the origin (0, 0). When the movement amount of the host vehicle can be observed, the movement amount ΔV of the host vehicle from time t to time t + 1 (refer to Figure 10 ) is obvious. Here, the movement prediction unit 115 sets the prior distribution and the likelihood function for the moving object to be predicted, and obtains the posterior distribution. By maximizing the posterior probability, it is possible to correspond to the shape of the complex posterior distribution. Based on the predicted posterior distribution, the existence range of the tracking object is specified. The moving object tracking unit 116 performs point cloud matching within the range of the posterior distribution. When the point clouds match each other, the moving object tracking unit 116 regards the matching point clouds as the point clouds of the same object and tracks them. In addition, the moving object tracking unit 116 calculates the accurate movement amount (relative distance) and posture (rotation angle) of the moving object relative to the host vehicle. The calculated movement amount and posture are the observed values.

[0169] In Figure 9 , the movement prediction unit 115 sets the prior distribution P t (A) for the vehicle in front of the host vehicle (moving object A) and the prior distribution P t (B) for the vehicle in the oncoming lane (moving object B).

[0170] At the prediction initial stage (t = 0), the movement prediction unit 115 sets the prior distribution P t (A) and the likelihood function P t (X|A) according to the attributes of the moving object A.

[0171] At the prediction initial stage (t = 0), the movement prediction unit 115 sets the prior distribution P t (B) and the likelihood function P t (X|B) according to the attributes of the moving object B.

[0172] The prior distribution and the likelihood function based on the attributes of the moving object will be described later.

[0173] In Figure 10 , the movement prediction unit 115 uses the prior distribution P t(A) and the likelihood function P t (X|A), calculates the position of the moving object A at time t+1 (predicted value  t+1 ) and the posterior distribution P t (A|X).

[0174] In addition, the movement prediction unit 115 uses the prior distribution P t (B) and the likelihood function P t (X|B), calculates the position of the moving object B at time t+1 (predicted value B̂ t+1 ) and the posterior distribution P t (B|X).

[0175] Based on Figure 11 and Figure 12 , the prior distribution and the likelihood function based on the attributes of the moving object are described. The shading indicates the prior distribution.

[0176] Figure 11 Visually shows the prior distribution that varies according to the category of the moving object.

[0177] Figure 12 Visually shows the prior distribution that varies according to the moving direction (heading) of the moving object.

[0178] Similarly, the likelihood function also varies according to the category, the moving direction, etc.

[0179] ***Description of the embodiment***

[0180] The prediction tracking device 100 only needs to be able to obtain the sensor data set from the sensor group 180 of the host vehicle 210, and may also be provided outside the host vehicle 210.

[0181] The sensor group 180 only needs to be able to perform measurement on the host vehicle 210 and the periphery of the host vehicle 210, and may also be provided outside the host vehicle 210.

[0182] As Figure 13 shown, the sensor group 180 may also include sensors such as the camera 184, the millimeter wave radar 185, and the sonar 186.

[0183] The camera 184 captures the periphery of the host vehicle 210 and outputs image data. The image data represents an image capturing the periphery of the host vehicle 210. The camera 184 is, for example, a stereo camera.

[0184] The millimeter wave radar 185 outputs reflected distance data using millimeter waves instead of lasers. The reflected distance data of the millimeter wave radar 185 represents the distance for each location where the millimeter wave is reflected.

[0185] The sonar 186 uses sound waves instead of laser to output reflected distance data. The reflected distance data of the sonar 186 represents the distance for each location where the sound wave is reflected.

[0186] By using the image data and the reflected distance data, the estimation of the own position can be performed with high precision.

[0187] It is also possible to estimate the own position using only the image data. For example, the own position estimation unit 112 uses the image data obtained by the stereo camera to estimate the own position. Alternatively, the own position estimation unit 112 measures the distance from the feature points in the image through Visual SLAM and estimates the own position.

[0188] By using the image data and the reflected distance data, the extraction accuracy of the point cloud of the moving object and the discrimination accuracy of the attributes of the moving object can be improved.

[0189] It is also possible to use a prediction filter different from the Kalman filter and Bayesian estimation. For example, a well-known filter such as a particle filter can also be used. In addition, it is also possible to switch and use multiple filters by the IMM method. IMM is the abbreviation of Interacting Multiple Model.

[0190] It is also possible to use an algorithm other than the ICP algorithm for point cloud matching. For example, the NDT algorithm can also be used. NDT is the abbreviation of Normal Distributions Transform.

[0191] The moving object tracking unit 116 can also use the tracking duration to evaluate the accuracy of tracking. In addition, in order to evaluate the accuracy of tracking, in addition to the tracking duration, the tracking interruption time can also be used. The tracking duration is an index for confirming how much time the tracked object has been accurately tracked. The tracking interruption time is, for example, the time of the tracking process that is repeated from when the tracked object hides in the shadow until it is observed again.

[0192] The moving object tracking unit 116 can also use the tracking duration to obtain the probability that the predicted position of the tracked object exists for the tracked object.

[0193] ***Effects of Embodiment 1***

[0194] The prediction tracking device 100 can set the prior distribution of the moving object in consideration of the category (such as truck, bus, passenger car, motorcycle, or pedestrian, etc.) and the moving direction (heading) obtained from the recognition result of the moving object. Thereby, the prediction accuracy when tracking the moving object is improved.

[0195] The prediction tracking device 100 performs the matching of point clouds based on the ICP algorithm only within the range of the predicted posterior distribution. Thereby, the reduction of the computational cost and the accurate measurement of the movement amount of other vehicles are achieved.

[0196] When the prediction tracking device 100 obtains the posterior distribution to maximize the posterior probability, it uses approximation methods such as Laplace approximation or variational Bayesian method. Thereby, the prediction accuracy when tracking a moving object is improved.

[0197] The prediction tracking device 100 can also detect (extraction of point clouds), identify (discrimination of attributes), and track a moving object only using the point cloud data in the sensor data group. That is, it can also detect, identify, and track a moving object only using the LiDAR 181 in the sensor group 180.

[0198] Embodiment 2

[0199] Regarding the method using sensor fusion, according to Figure 14 , the aspects different from Embodiment 1 will be mainly described.

[0200] ***Description of the structure***

[0201] According to Figure 14 , the structure of the prediction tracking device 100 will be described.

[0202] The prediction tracking device 100 further includes a sensor fusion unit 117.

[0203] The prediction tracking program also causes the computer to function as the sensor fusion unit 117.

[0204] ***Description of the operation***

[0205] The sensor fusion unit 117 performs sensor fusion using two or more types of sensor data included in the sensor data group.

[0206] As the main methods for sensor fusion, there are methods for fusing RAW data, methods for fusing intermediate data, and methods for fusing post-detection data. Any method uses methods such as deep learning, and can perform the extraction of the point cloud of a moving object and the discrimination of the attributes of a moving object with high accuracy compared to the case of using only one type of sensor data alone.

[0207] In sensor fusion, there are types such as early fusion, cross fusion, and late fusion.

[0208] As combinations of sensors in sensor fusion, various combinations such as the camera 184 and the LiDAR 181, the LiDAR 181 and the millimeter-wave radar 185, or the camera 184 and the millimeter-wave radar 185 can be considered.

[0209] The self-position estimation unit 112 estimates the self-position using the data obtained by sensor fusion for two or more types of sensor data.

[0210] The moving object detection unit 113 calculates the relative position of the detected object based on the data obtained by sensor fusion for two or more types of sensor data including point cloud data.

[0211] ***Effects of Embodiment 2***

[0212] By using sensor fusion, it is possible to detect, identify, and track moving objects with higher accuracy.

[0213] ***Supplement to the Embodiment***

[0214] According to Figure 15 , the hardware structure of the prediction tracking device 100 will be described.

[0215] The prediction tracking device 100 includes a processing circuit 109.

[0216] The processing circuit 109 is the hardware that implements the sensor data acquisition unit 111, the self-position estimation unit 112, the moving object detection unit 113, the moving object identification unit 114, the movement prediction unit 115, the moving object tracking unit 116, and the sensor fusion unit 117.

[0217] The processing circuit 109 can be dedicated hardware or a processor 101 that executes a program stored in the memory 102.

[0218] When the processing circuit 109 is dedicated hardware, the processing circuit 109 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0219] ASIC is an abbreviation for Application Specific Integrated Circuit.

[0220] FPGA is an abbreviation for Field Programmable Gate Array.

[0221] The prediction tracking device 100 may also include a plurality of processing circuits instead of the processing circuit 109. The plurality of processing circuits share the functions of the processing circuit 109.

[0222] In the processing circuit 109, some functions may be implemented by dedicated hardware, and the remaining functions may be implemented by software or firmware.

[0223] In this way, the functions of the prediction and tracking device 100 can be implemented by hardware, software, firmware, or a combination thereof.

[0224] Each embodiment is an illustration of a preferred mode and is not intended to limit the technical scope of the present invention. Each embodiment can be implemented partially or in combination with other embodiments. The steps described using flowcharts and the like can also be changed appropriately.

[0225] The "section" as an element of the prediction and tracking device 100 can also be replaced with "processing" or "process".

[0226] Reference Numeral Explanation

[0227] 100: Prediction and tracking device; 101: Processor; 102: Memory; 103: Auxiliary storage device; 104: Communication device; 105: Input / output interface; 109: Processing circuit; 111: Sensor data acquisition section; 112: Own position estimation section; 113: Moving object detection section; 114: Moving object identification section; 115: Movement prediction section; 116: Moving object tracking section; 117: Sensor fusion section; 180: Sensor group; 181: LiDAR; 182: GPS; 183: Speedometer; 184: Camera; 185: Millimeter-wave radar; 186: Sonar; 190: Storage section; 200: Prediction and tracking system; 210: Own vehicle.

Claims

1. A prediction and tracking device, wherein, the prediction and tracking device has: a moving object detection unit that extracts point clouds representing moving objects existing around the moving body from point cloud data obtained by a LiDAR that measures around the moving body; a moving object tracking unit that extracts point clouds within the range of the posterior distribution for the tracking object of the moving object being tracked from the point cloud data, and performs matching between the extracted point clouds and the point clouds of the moving objects; a moving object recognition unit that determines the attributes of the moving objects based on the point clouds of the moving objects; and a movement prediction unit that sets a prior distribution and a likelihood function for the moving object based on the attributes of the moving object, and calculates the posterior distribution for the moving object using the prior distribution and the likelihood function for the moving object, wherein the prior distribution and the likelihood function for the moving objects having different attributes are different.

2. The prediction and tracking device according to claim 1, wherein, the moving object recognition unit determines the category of the moving object, and the movement prediction unit sets a prior distribution and a likelihood function for the moving object based on the category of the moving object.

3. The prediction and tracking device according to claim 1, wherein, the moving object recognition unit determines the moving direction of the moving object, and the movement prediction unit sets a prior distribution and a likelihood function for the moving object based on the moving direction of the moving object.

4. The prediction and tracking device according to claim 2, wherein, the moving object recognition unit determines the moving direction of the moving object, and the movement prediction unit sets a prior distribution and a likelihood function for the moving object based on the moving direction of the moving object.

5. The prediction and tracking device according to any one of claims 1 to 4, wherein, when the movement prediction unit approximates the posterior probability in maximizing the posterior probability, it uses an approximation estimation method.

6. A prediction and tracking method, wherein, a moving object detection unit extracts point clouds representing moving objects existing around the moving body from point cloud data obtained by a LiDAR that measures around the moving body, a moving object tracking unit extracts point clouds within the range of the posterior distribution for the tracking object of the moving object being tracked from the point cloud data, and performs matching between the extracted point clouds and the point clouds of the moving objects, a moving object recognition unit determines the attributes of the moving objects based on the point clouds of the moving objects, a movement prediction unit sets a prior distribution and a likelihood function for the moving object based on the attributes of the moving object, and calculates the posterior distribution for the moving object using the prior distribution and the likelihood function for the moving object, wherein the prior distribution and the likelihood function for the moving objects having different attributes are different.

7. A computer-readable recording medium storing a prediction and tracking program, the prediction and tracking program for causing a computer to execute the following processing: Moving object detection process, which extracts point clouds representing moving objects existing around the moving body from the point cloud data obtained by a LiDAR that measures the periphery of the moving body; Moving object tracking process, which extracts point clouds within the range of the posterior distribution for the tracked object, which is a moving object being tracked, from the point cloud data, and matches the extracted point clouds with the point clouds of the moving object; Moving object recognition process, which discriminates the attributes of the moving object based on the point clouds of the moving object; And Moving prediction process, which sets the prior distribution and likelihood function for the moving object according to the attributes of the moving object, and calculates the posterior distribution for the moving object using the prior distribution and likelihood function for the moving object, The prior distribution and the likelihood function for the moving objects with different attributes are different.

Images