Object tracking device and object tracking program

Abstract

To provide an object tracking device capable of suppressing degradation of robustness for object tracking, and an object tracking program.SOLUTION: An object tracking device 20 tracks a tracking target object by predicting a position and reflection intensity of a distance measurement point, and associating the distance measurement point with a predicted distance measurement point based on the position and reflection intensity of the distance measurement point and the position and reflection intensity of the predicted distance measurement point, and updating the position and reflection intensity of the predicted distance measurement point based on the distance measurement point and the predicted distance measurement point corresponding to each other. Also, the object tracking device 20 predicts the position and reflection intensity of the distance measurement point based on the updated position and reflection intensity of the predicted distance measurement point.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an object tracking device and an object tracking program. [Background technology]

[0002] Conventionally, as described in Non-Patent Document 1, an algorithm for associating point groups using reflection intensity information and geometric information is known. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] J. Park, Q.-Y. Zhou and V. Koltun, "Colored Point Cloud Registration Revisited," 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017, pp. 143-152, doi:10.1109 / ICCV.2017.25. Summary of the Invention [Problem to be solved by the invention]

[0004] In the algorithm of Non-Patent Document 1, instantaneous reflection intensity information and geometric information are used, so the accuracy of matching of point clouds decreases due to noise such as variations in reflection intensity information and geometric information and variations in reflection intensity caused by light blocking, etc. Therefore, the robustness of matching of point clouds is low in the algorithm of Non-Patent Document 1. For this reason, the algorithm of Non-Patent Document 1 cannot be simply used for object tracking.

[0005] The present disclosure aims to provide an object tracking device and an object tracking program that suppress a decrease in robustness in object tracking. [Means for solving the problem]

[0006] The invention described in claim 1 is an object tracking device comprising an acquisition unit (S100) that acquires position data of a distance measurement point (Pm) detected by a distance measurement sensor (12) and reflection data which is data relating to the reflection intensity of a search wave of the distance measurement sensor reflected at the distance measurement point, an observation unit (S102) that extracts distance measurement points that contribute to a value relating to the shape of a tracked object (50) based on the position data and the reflection data and calculates values ​​relating to the positions and reflection intensity for the extracted distance measurement points, a prediction unit (S108) that predicts the values ​​relating to the positions and reflection intensity for the distance measurement points extracted by the observation unit, and a prediction unit (S109) that predicts the values ​​relating to the positions and reflection intensity for the distance measurement points extracted by the observation unit. The object tracking device includes a correspondence calculation unit (S110) that matches the ranging points extracted by the observation unit with the predicted ranging points based on values ​​related to the position and reflection intensity of the predicted ranging points (Pp), which are ranging points predicted by the prediction unit, and an update unit (S112) that tracks the tracked object by updating the values ​​related to the position and reflection intensity of the predicted ranging points based on the ranging points extracted by the observation units and the predicted ranging points that correspond to each other, and the prediction unit predicts the values ​​related to the position and reflection intensity of the ranging points extracted by the observation unit based on the values ​​related to the position and reflection intensity of the predicted ranging points updated by the update unit.

[0007] The invention described in claim 10 is an object tracking device, comprising: an acquisition unit (S100) for acquiring position data of distance measurement points (Pm) detected by an image sensor (40) and brightness data which is data relating to the brightness of the distance measurement points in an image captured by the image sensor; an observation unit (S102) for extracting distance measurement points which contribute to a value relating to the shape of a tracked object (50) based on the position data and brightness data, and calculating values ​​relating to the positions and brightness of the extracted distance measurement points; a prediction unit (S108) for predicting values ​​relating to the positions and brightness of the distance measurement points extracted by the observation unit; the object tracking device includes a correspondence calculation unit (S110) that matches the ranging points extracted by the observation unit with the predicted ranging points based on the corresponding values ​​and values ​​related to position and brightness for the predicted ranging points (Pp), which are ranging points predicted by the prediction unit, and an update unit (S112) that tracks the tracked object by updating the values ​​related to position and brightness for the predicted ranging points based on the ranging points extracted by the observation unit and the predicted ranging points that correspond to each other, and the prediction unit predicts the values ​​related to position and brightness for the ranging points extracted by the observation unit based on the values ​​related to position and brightness for the predicted ranging points updated by the update unit.

[0008] Furthermore, the invention described in claim 11 is an object tracking device including an acquisition unit (S100) that acquires position data of a distance measurement point (Pm) detected by a distance measurement sensor (12) and velocity data that is data related to Doppler velocity for the distance measurement point, an observation unit (S102) that extracts distance measurement points that contribute to a value related to the shape of a tracked object (50) based on the position data velocity data and calculates values ​​related to position and Doppler velocity for the extracted distance measurement points, a prediction unit (S108) that predicts values ​​related to position and Doppler velocity for the distance measurement points extracted by the observation unit, and a prediction unit (S109) that predicts values ​​related to position and Doppler velocity for the distance measurement points extracted by the observation unit. The object tracking device is equipped with a correspondence calculation unit (S110) that matches the ranging points extracted by the observation unit with the predicted ranging points based on values ​​related to the position and Doppler velocity for the predicted ranging points (Pp), which are predicted ranging points, and an update unit (S112) that tracks the tracked object by updating the values ​​related to the position and Doppler velocity for the predicted ranging points based on the ranging points extracted by the observation units and the predicted ranging points that correspond to each other, and the prediction unit predicts the values ​​related to the position and Doppler velocity for the ranging points extracted by the observation unit based on the values ​​related to the position and Doppler velocity for the predicted ranging points updated by the update unit.

[0009] The invention described in claim 12 is an object tracking program, comprising an object tracking device including an acquisition unit (S100) that acquires position data of a distance measurement point (Pm) detected by a distance measurement sensor (12) and reflection data which is data relating to the reflection intensity of a search wave of the distance measurement sensor reflected at the distance measurement point, an observation unit (S102) that extracts distance measurement points which contribute to a value relating to the shape of the tracked object (50) based on the position data and the reflection data and calculates values ​​relating to the positions and reflection intensity for the extracted distance measurement points, a prediction unit (S108) that predicts values ​​relating to the positions and reflection intensity for the distance measurement points extracted by the observation unit, and a prediction unit (S109) that predicts values ​​relating to the positions and reflection intensity for the distance measurement points extracted by the observation unit. the object tracking program functions as a correspondence calculation unit (S110) that matches the ranging points extracted by the observation unit with the predicted ranging points based on the corresponding values ​​and values ​​related to the position and reflection intensity of the predicted ranging points (Pp), which are ranging points predicted by the prediction unit, and an update unit (S112) that tracks the tracked object by updating the values ​​related to the position and reflection intensity of the predicted ranging points based on the ranging points and predicted ranging points extracted by the observation units that correspond to each other, and the prediction unit predicts the values ​​related to the position and reflection intensity of the ranging points extracted by the observation unit based on the values ​​related to the position and reflection intensity of the predicted ranging points updated by the update unit.

[0010] Furthermore, the invention described in claim 13 is an object tracking program, comprising an object tracking device including an acquisition unit (S100) that acquires position data of distance measurement points (Pm) detected by an image sensor (40) and luminance data which is data relating to the luminance of the distance measurement points in an image captured by the image sensor, an observation unit (S102) that extracts distance measurement points that contribute to a value relating to the shape of a tracked object (50) based on the position data and luminance data, and calculates values ​​relating to the positions and luminance of the extracted distance measurement points, a prediction unit (S108) that predicts the values ​​relating to the positions and luminance of the distance measurement points extracted by the observation unit, and a prediction unit (S109) that predicts the values ​​relating to the positions and luminance of the distance measurement points extracted by the observation unit. The object tracking program has a correspondence calculation unit (S110) that associates the ranging points extracted by the observation unit with the predicted ranging points based on the value obtained by the correspondence calculation unit and values ​​related to the position and brightness of the predicted ranging points (Pp), which are ranging points predicted by the prediction unit, and an update unit (S112) that tracks the tracked object by updating the values ​​related to the position and brightness of the predicted ranging points based on the ranging points and predicted ranging points extracted by the observation unit that correspond to each other, and the prediction unit predicts the values ​​related to the position and brightness of the ranging points extracted by the observation unit based on the values ​​related to the position and brightness of the predicted ranging points updated by the update unit.

[0011] The invention described in claim 14 is an object tracking program, comprising an object tracking device, an acquisition unit (S100) that acquires position data of a distance measurement point (Pm) detected by a distance measurement sensor (12) and velocity data that is data related to Doppler velocity for the distance measurement point, an observation unit (S102) that extracts distance measurement points that contribute to a value related to the shape of a tracked object (50) based on the position data and velocity data, and calculates values ​​related to position and Doppler velocity for the extracted distance measurement points, a prediction unit (S108) that predicts values ​​related to position and Doppler velocity for the distance measurement points extracted by the observation unit, and a prediction unit (S109) that predicts values ​​related to position and Doppler velocity for the distance measurement points extracted by the observation unit and the values ​​related to position and Doppler velocity for the distance measurement points extracted by the observation unit. The object tracking program functions as a correspondence calculation unit (S110) that associates the ranging points extracted by the observation unit with the predicted ranging points based on values ​​related to the position and Doppler speed for the predicted ranging points (Pp), which are ranging points predicted by the observation unit, and an update unit (S112) that tracks the tracked object by updating values ​​related to the position and Doppler speed for the predicted ranging points based on the ranging points extracted by the observation units and the predicted ranging points that correspond to each other, and the prediction unit predicts values ​​related to the position and Doppler speed for the ranging points extracted by the observation unit based on the values ​​related to the position and Doppler speed for the predicted ranging points updated by the update unit.

[0012] This allows the orientation of the tracked object to be taken into consideration when matching between the ranging points and the predicted ranging points. In addition, since not only the instantaneous information but also the predicted information is used, the accuracy of matching between the point clouds is prevented from decreasing due to noise such as variations in the point clouds, compared to when only the instantaneous information is used. Therefore, the robustness of object tracking is prevented from decreasing.

[0013] The reference symbols in parentheses attached to each component indicate an example of the correspondence between the component and specific components described in the embodiments described below. [Brief description of the drawings]

[0014] [Figure 1]FIG. 1 is a configuration diagram of a vehicle in which the object tracking device according to the first and third embodiments is used. [Diagram 2] 11 is a flowchart showing the processing of the object tracking device. [Diagram 3] FIG. 2 is a diagram showing distance measurement points of a tracked object by the object tracking device. [Figure 4] FIG. 13 is a diagram showing calculation of a convex hull of distance measurement points. [Diagram 5] 6 is a diagram showing extraction of distance measurement points that contribute to a value related to the shape of a tracked object by an object tracking device; [Figure 6] 4A and 4B are diagrams for explaining initial values ​​for an object to be tracked by the object tracking device; [Figure 7] FIG. 4 is a diagram showing a shape function indicating the shape of a tracked object. [Figure 8] Histogram of the average initial reflection intensity versus the initial relative angle of the tracked object. [Figure 9] 5A and 5B are diagrams for explaining prediction information about a tracked object by the object tracking device. [Figure 10] 11A and 11B are diagrams for explaining correction of prediction information by the object tracking device. [Figure 11] A diagram showing the case where the incident angle of the search wave from the vehicle's distance measurement sensor is π / 2. [Figure 12] FIG. 4 is a diagram showing the relationship between the incident angle and the reflection intensity. [Figure 13] 5A and 5B are diagrams for explaining updated prediction information for a tracked object by the object tracking device; [Figure 14] FIG. 13 is a diagram showing point group matching by object tracking in a comparative example. [Figure 15] FIG. 13 is a diagram showing point group matching by object tracking in a comparative example. [Figure 16] FIG. 13 is a diagram showing point group matching performed by the object tracking device. [Figure 17] 10A and 10B are diagrams for explaining extraction of distance measurement points that contribute to values ​​related to the shape of a tracked object by an object tracking device according to a modified example. [Figure 18] FIG. 11 is a configuration diagram of a vehicle in which an object tracking device according to a second embodiment is used. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, embodiments will be described with reference to the drawings. In the following embodiments, the same or equivalent parts are denoted by the same reference numerals, and the description thereof will be omitted.

[0016] In the object tracking device of the present embodiment, the degradation of robustness in object tracking is suppressed. For example, the object tracking device is used in a vehicle. First, the vehicle will be described.

[0017] 1, a vehicle 10 includes a distance measurement sensor 12, an IMU 14, a wheel speed sensor 16, an object tracking device 20, and a vehicle ECU 30. Note that IMU is an abbreviation for Inertial Measurement Unit, and ECU is an abbreviation for Electronic Control Unit.

[0018] The distance measuring sensor 12 includes a radar, LiDAR, sonar, etc., using radio waves such as millimeter waves. Thus, the distance measuring sensor 12 detects a point cloud consisting of at least one distance measuring point Pm, which is a point for each position on the tracked object 50, using a search wave such as an electromagnetic wave and an ultrasonic wave. The distance measuring sensor 12 also generates position data and reflection data for a predetermined period for the detected distance measuring point Pm. Furthermore, the distance measuring sensor 12 outputs the generated position data and reflection data to the object tracking device 20 described later via an in-vehicle LAN 18 such as a CAN. The tracked object 50 is an object tracked by the object tracking device 20 described later, for example, a preceding vehicle with respect to the vehicle 10. The reflection data is data related to the reflection intensity of the search wave of the distance measuring sensor 12 reflected at the distance measuring point Pm. LiDAR is an abbreviation for Light Detection and Ranging / Laser Imaging Detection and Ranging. CAN is an abbreviation for Controller Area Network. LAN is an abbreviation for Local Area Network.

[0019] The IMU 14 has a six-axis inertial sensor composed of, for example, a three-axis gyro sensor and a three-axis acceleration sensor. With this, the IMU 14 detects the vehicle attitude Ac. The IMU 14 also outputs a signal corresponding to the detected vehicle attitude Ac to the vehicle ECU 30 described later via the in-vehicle LAN 18. The vehicle attitude Ac is the attitude of the vehicle 10, and is, for example, the roll angle, pitch angle, and yaw angle of the vehicle 10.

[0020] The wheel speed sensor 16 detects the rotation speed of the wheels of the vehicle 10. In this way, the wheel speed sensor 16 detects the vehicle speed Vc. Furthermore, the wheel speed sensor 16 outputs a signal corresponding to the detected vehicle speed Vc to the vehicle ECU 30 (described later) via the in-vehicle LAN 18. The vehicle speed Vc is the speed of the vehicle 10.

[0021] The object tracking device 20 is mainly composed of a microcomputer and includes a CPU, a ROM, a flash memory, a RAM, an I / O, a communication interface, and a bus line connecting these components. The object tracking device 20 executes a program stored in the ROM of the object tracking device 20. As a result, the object tracking device 20 tracks objects around the vehicle 10 based on data from the distance measurement sensor 12. The object tracking device 20 also calculates tracking information Ic, which is information about the tracked object 50. The object tracking device 20 then outputs the calculated tracking information Ic to the vehicle ECU 30, which will be described later, via the in-vehicle LAN 18.

[0022] The vehicle ECU 30 is mainly composed of a microcomputer and includes a CPU, a ROM, a flash memory, a RAM, an I / O, a communication interface, and a bus line connecting these components. The vehicle ECU 30 executes a program stored in the ROM of the vehicle ECU 30. As a result, the vehicle ECU 30 performs driving assistance such as ACC and automatic driving of the vehicle 10 based on the vehicle attitude Ac, the vehicle speed Vc, and the tracking information Ic. ACC is an abbreviation for Adaptive Cruise Control.

[0023] The vehicle 10 is configured as described above. Next, details of object tracking by the execution of the program of the object tracking device 20 will be described with reference to the flowchart of Fig. 2 and Figs. 3 to 13. The program of the object tracking device 20 is executed, for example, when the ignition or power of the vehicle 10 is turned on. Furthermore, the period of a series of operations from when the processing of step S100 of the object tracking device 20 is started to when the processing of step S100 is returned is defined as the control cycle of the object tracking device 20.

[0024] As shown in the flowchart of FIG. 2, the object tracking device 20 acquires position data and reflection data from the distance measurement sensor 12 in step S100.

[0025] Next, in step S102, the object tracking device 20 extracts distance measurement points Pm that contribute to values ​​related to the shape of the tracked object 50, which will be described later, based on the position data and reflection data acquired in step S100. In this way, the object tracking device 20 calculates current observation information Io(n), where n is a natural number and indicates the number of times the tracked object 50 has been tracked.

[0026] Specifically, the object tracking device 20 removes from the position data objects different from the tracked object 50, such as distance measurement points Pm on a road surface or the like. The object tracking device 20 also calculates the distances between the remaining distance measurement points Pm from the positions of the remaining distance measurement points Pm. Furthermore, the object tracking device 20 clusters the remaining distance measurement points Pm based on the calculated distances. As a result, the object tracking device 20 groups similar points among the remaining distance measurement points Pm.

[0027] Here, the distance measurement points Pm detected by the distance measurement sensor 12 contain noise in the shape calculation of the tracked object 50 described below due to accidental reflection of the search wave, etc. For example, as shown in Fig. 3, when the tracked object 50 is a preceding vehicle, the distance measurement points Pm inside the preceding vehicle when viewed from above do not contribute to a value related to the shape of the tracked object 50, and therefore become noise in the shape calculation of the tracked object 50. Moreover, the distance measurement points Pm on the outer periphery when viewed from above are points that contribute to a value related to the shape of the tracked object 50.

[0028] For this reason, the object tracking device 20 extracts, from the grouped distance points Pm, distance points Pm that contribute to values ​​related to the shape of the tracked object 50. In this way, the object tracking device 20 removes noise in the calculation of the shape of the tracked object 50 from the grouped distance points Pm.

[0029] For example, the object tracking device 20 calculates a convex hull for the grouped distance measurement points Pm as shown in Fig. 4. Note that the convex hull is the smallest convex polygon that includes all the given points.

[0030] Furthermore, here, the distance from a ranging point Pm that does not contribute to values ​​related to the shape of the tracked object 50, for example, a ranging point Pm inside the preceding vehicle when viewed from above, to an edge of the convex hull, is smaller than the distance from a ranging point Pm on the outer periphery of the preceding vehicle to an edge of the convex hull.

[0031] Therefore, the object tracking device 20 calculates the distance from each side of the calculated convex hull to each ranging point Pm. Furthermore, the object tracking device 20 removes ranging points Pm for which the calculated distances are less than a predetermined distance. As a result, the object tracking device 20 removes ranging points Pm that do not contribute to the value related to the shape of the tracked object 50, as shown in FIG. 5, here, ranging points Pm inside the preceding vehicle when the preceding vehicle is viewed from above. Furthermore, the object tracking device 20 extracts ranging points Pm for which the calculated distances are equal to or greater than a predetermined distance. As a result, the object tracking device 20 extracts ranging points Pm that contribute to the value related to the shape of the tracked object 50, here, ranging points Pm on the outer periphery when the preceding vehicle is viewed from above. Then, the object tracking device 20 calculates the positions and reflection intensities of these extracted ranging points Pm as current observation information Io(n). The above-mentioned predetermined distance is set by experiments, simulations, etc. so that distance measurement points Pm that do not contribute to values ​​related to the shape of the tracked object 50 are removed and distance measurement points Pm that contribute to values ​​related to the shape of the tracked object 50 are extracted.

[0032] 2, next, in step S104, the object tracking device 20 judges whether or not it is the first time that the tracked object 50 is tracked. For example, the object tracking device 20 judges whether or not it is the first time that the tracked object 50 is tracked, depending on whether or not updated prediction information Ip_C for the tracked object 50 described later has been calculated.

[0033] Then, when the update prediction information Ip_C has been calculated, as will be described later, the tracked object 50 has been tracked, so this is not the first time the tracked object 50 has been tracked. Therefore, at this time, the processing of the object tracking device 20 proceeds to step S108. On the other hand, when the update prediction information Ip_C has not been calculated, the tracked object 50 has not been tracked, so this is the first time the tracked object 50 has been tracked. Therefore, at this time, the processing of the object tracking device 20 proceeds to step S106.

[0034] In step S106 following step S104, since tracking of the tracked object 50 is the first time, the object tracking device 20 calculates an initial value Ip(0) in order to calculate prediction information Ip described below. Specifically, the object tracking device 20 calculates the following parameters as initial values ​​Ip(0) based on the position data and reflection data of the distance measurement point Pm extracted in step S102 during a predetermined period including the time acquired in step S100.

[0035] Here, as shown in FIG. 6, a Cartesian coordinate system based on the position of the distance measuring sensor 12 is defined as an absolute coordinate system Σ. The X-axis, Y-axis, and Z-axis in the absolute coordinate system Σ are mutually orthogonal. Furthermore, this absolute coordinate system Σ is represented as a right-handed system. The front of the vehicle 10 is defined as the positive direction of the X-axis. Furthermore, the left side of the vehicle 10 is defined as the positive direction of the Y-axis. Therefore, the positive direction of the Z-axis corresponds to the sky.

[0036] Then, the object tracking device 20 calculates, for example, a rectangle of a convex hull for the distance measurement point Pm extracted in step S102. The object tracking device 20 also calculates an initial x coordinate xp(0) and an initial y coordinate yp(0) of the center position Op of this calculated rectangle. Furthermore, the object tracking device 20 calculates an initial object length Lp(0) and an initial object width Wp(0), which are the lengths of this calculated rectangle.

[0037] Furthermore, the object tracking device 20 divides the amount of change in the center position Op during a predetermined period including the time acquired in step S100 by the predetermined period. In this way, the object tracking device 20 calculates the initial object speed Vp(0), which is the speed of the rectangle calculated above.

[0038] Furthermore, the object tracking device 20 calculates an initial object yaw angle ψp(0) which is the yaw angle of the calculated rectangle. The object tracking device 20 also divides the amount of change in the object yaw angle ψp during a predetermined period including the time acquired in step S100 by the predetermined period. In this way, the object tracking device 20 calculates an initial object yaw angular velocity ωp(0) which is the yaw angular velocity of the calculated rectangle. The yaw angle is a rotation angle around the Z axis in the absolute coordinate system Σ.

[0039] The object tracking device 20 also calculates an initial center distance rp(0) from the calculated center position Op to the distance measurement point Pm extracted in step S102. Furthermore, the object tracking device 20 calculates an initial relative angle θp(0) of the distance measurement point Pm extracted in step S102 with respect to the front of the tracked object 50. Note that the relative angle θp is the angle of the distance measurement point Pm with respect to the front of the tracked object 50, and is an angle around the Z axis in the absolute coordinate system Σ.

[0040] Furthermore, the object tracking device 20 calculates an initial shape function rp(0)=f(θp(0)) indicating the shape of the tracked object 50 from the calculated initial center distance rp(0) and initial relative angle θp(0). The shape function is, for example, a function indicating a quadrangle such as a rectangle. Furthermore, the shape function is not limited to a function indicating a quadrangle, and may be an elliptical function, a star-shaped convex polygon function, or a B-spline function as shown in FIG. 7.

[0041] Furthermore, the object tracking device 20 calculates an initial reflection function Ep(0)=g(θp(0)) indicating the initial reflection state of the tracked object 50 based on the initial relative angle θp(0) calculated above and the initial reflection intensity Eo(0) of the distance measurement point Pm extracted in step S102. The initial reflection function Ep(0)=g(θp(0)) is expressed, for example, as shown in Fig. 8, by a histogram or the like using the partitions of the initial relative angle θp(0) and the average value such as the moving average, arithmetic average or weighted average of the initial reflection intensity Eo(0) for each partition. In Fig. 8, the average value of the initial reflection intensity Eo(0) is indicated as Ep(0)_ave.

[0042] From the above, the object tracking device 20 calculates the initial x coordinate xp(0), initial y coordinate yp(0), initial object length Lp(0), initial object width Wp(0) and initial object speed Vp(0) of the center position Op as initial values ​​Ip(0). Furthermore, the object tracking device 20 calculates the initial object yaw angle ψp(0), initial object yaw angular velocity ωp(0), initial center distance rp(0) and initial relative angle θp(0) as initial values ​​Ip(0). Furthermore, the object tracking device 20 calculates the initial shape function rp(0)=f(θ(0)) and the initial reflection function Ep(0)=g(θp(0)) as initial values ​​Ip(0). After these parameters are calculated, the process of the object tracking device 20 proceeds to step S108.

[0043] Returning to the flowchart of Fig. 2, in step S108, when tracking of the tracked object 50 is the first time, n is 1, so the object tracking device 20 calculates predicted information Ip(1) based on the initial value Ip(0) calculated in step S106. Furthermore, when tracking of the tracked object 50 is not the first time, the object tracking device 20 calculates current predicted information Ip(n) based on previously updated predicted information Ip_C(n-1). Note that the updated predicted information Ip_C is the updated predicted information Ip, and is calculated in step S112 described later.

[0044] Specifically, the object tracking device 20 assumes that the tracked object 50 moves at a constant speed and constant acceleration. Then, as shown in Fig. 9, the object tracking device 20 calculates a current predicted x coordinate xp(n) of the center position Op from this motion model and the previously updated x coordinate xp_C(n-1) of the updated center position Op_C. The object tracking device 20 also calculates a current predicted y coordinate yp(n) of the center position Op from the motion model and the previously updated y coordinate yp_C(n-1) of the updated center position Op_C.

[0045] Furthermore, the object tracking device 20 calculates a current predicted object length Lp(n) from the motion model and the previously updated object length Lp_C(n-1). Also, the object tracking device 20 calculates a current predicted object width Wp(n) from the motion model and the previously updated object width Wp_C(n-1).

[0046] Furthermore, the object tracking device 20 calculates a current predicted object speed Vp(n) from the motion model and the previously updated object speed Vp_C(n-1). The object tracking device 20 also calculates a current predicted object yaw angle ψp(n) from the motion model and the previously updated object yaw angle ψp_C(n-1). The object tracking device 20 also calculates a current predicted object yaw angular velocity ωp(n) from the motion model and the previously updated object yaw angular velocity ωp_C(n-1).

[0047] The object tracking device 20 also calculates a current predicted center distance rp(n) from the motion model and the previously updated center distance rp_C(n-1). The object tracking device 20 also calculates a current predicted relative angle θp(n) from the motion model and the previously updated relative angle θp_C(n-1).

[0048] Furthermore, the object tracking device 20 calculates a current predicted shape function rp(n)=f(θp(n)) from the current predicted center distance rp(n) and current predicted relative angle θp(n) calculated above. Note that the current predicted shape function rp(n)=f(θp(n)) is expressed by a B-spline function here. The current predicted shape function rp(n)=f(θp(n)) is not limited to being expressed by a B-spline function, and may be a function indicating a quadrangle such as a rectangle, an elliptical function, a star-shaped convex polygon function, or the like.

[0049] Furthermore, the object tracking device 20 calculates the current predicted reflection function Ep(n)=g(θp(n)) based on the previously updated prediction information Ip_C(n−1) as follows.

[0050] Here, the reflection intensity is inversely proportional to the kth power of the propagation distance due to attenuation of the probe wave, where k is a coefficient that varies depending on the size of the tracked object 50 and the diameter of the probe wave.

[0051] Therefore, the object tracking device 20 calculates a current predicted ranging point Pp(n) from the motion model and the previously updated predicted ranging point Pp_C(n-1). The object tracking device 20 also calculates a current predicted distance Dp(n) from the calculated current predicted ranging point Pp(n). The predicted distance Dp is the distance from the distance sensor 12 to the predicted ranging point Pp, and corresponds to the propagation distance of the search wave.

[0052] Furthermore, the object tracking device 20 calculates, for example, the area of ​​the convex hull for the distance measurement point Pm extracted in step S102. As a result, the object tracking device 20 calculates the size of the tracked object 50. When the calculated size of the tracked object 50 is equal to or smaller than the threshold, the object tracking device 20 sets k to, for example, 4. When the calculated size of the tracked object 50 is larger than the threshold, the object tracking device 20 sets k to, for example, 2. The threshold for the size of the tracked object 50 is set by experiment, simulation, or the like so that k suitable for the size of the tracked object 50 and the diameter of the search wave is calculated.

[0053] Here, the reflection intensity is proportional to the value of the angle between the tangent to the reflection point and the line connecting the reflection point and the distance measuring sensor 12. Furthermore, the value of this angle differs depending on the type of object.

[0054] Therefore, the object tracking device 20 calculates the current predicted incident angle αp(n) from the current predicted ranging point Pp(n) calculated above. The incident angle αp is the angle between the line connecting the predicted ranging point Pp and the ranging sensor 12 and the tangent to the predicted ranging point Pp.

[0055] Furthermore, the object tracking device 20 identifies the type of the tracked object 50 from the calculated current predicted shape function rp(n)=f(θp(n)) and the map. The map for identifying the type of the tracked object 50 is set by experiments, simulations, or the like so that the type of the tracked object 50 can be identified.

[0056] Furthermore, the object tracking device 20 calculates h(αp(n)), which is a value related to the current predicted incident angle αp(n), from the type of the tracked object 50 identified above, the calculated current predicted incident angle αp(n), and a map. Note that the map for calculating h(αp(n)) is set by experiments, simulations, etc., so as to increase the prediction accuracy of the reflection intensity.

[0057] Moreover, here, an error occurs in the reflection intensity depending on the attitude of the tracked object 50. For this reason, the object tracking device 20 calculates the current prediction variance σψ(n) of the current predicted object yaw angle ψp(n) calculated above.

[0058] Then, the object tracking device 20 substitutes the calculated current predicted distance Dp(n), k, h(αp(n)) and current prediction variance σψ(n) into the following relational expression (1). Furthermore, the object tracking device 20 substitutes the previous predicted distance Dp(n-1), h(αp(n-1)) calculated in the previous control cycle and the previous updated reflection intensity Ep_C(n-1) into the following relational expression (1). As a result, the object tracking device 20 calculates the current predicted reflection intensity Ep(n).

[0059]

number

[0060] Furthermore, the object tracking device 20 calculates a current predicted reflection function Ep(n)=g(θp(n)) from the current predicted reflection intensity Ep(n) and the current predicted relative angle θp(n) calculated above.

[0061] From the above, the object tracking device 20 calculates the current predicted x-coordinate xp(n) of the center position Op, the current predicted y-coordinate yp(n) of the center position Op, the current predicted object length Lp(n), and the current predicted object width Wp(n) as the current predicted information Ip(n). Furthermore, the object tracking device 20 calculates the current predicted object speed Vp(n), the current predicted object yaw angle ψp(n), the current predicted object yaw angular velocity ωp(n), the current predicted center distance rp(n), and the current predicted relative angle θp(n) as the current predicted information Ip(n). Furthermore, the object tracking device 20 calculates the current predicted shape function rp(n)=f(θp(n)) as the current predicted information Ip(n). Furthermore, the object tracking device 20 calculates the current predicted reflection intensity Ep(n) and the current predicted reflection function Ep(n)=g(θp(n)) as the current predicted information Ip(n).

[0062] Returning to the flowchart of FIG. 2, next, in step S110, the object tracking device 20 associates the current observation information Io(n) calculated in step S102 with the current prediction information Ip(n) calculated in step S108.

[0063] Specifically, the object tracking device 20 corrects the current predicted ranging point Pp(n) so that the current predicted ranging point Pp(n) of the current prediction information Ip(n) approaches the ranging point Pm of the current observation information Io(n). This makes it easier for the object tracking device 20 to associate the current observation information Io(n) with the current prediction information Ip(n).

[0064] For example, the object tracking device 20 uses ICP and the following relational expression (2) to calculate a rotation matrix R(n) and a translation matrix T(n) that bring the current predicted distance measurement point Pp(n) closest to the distance measurement point Pm. It should be noted that ICP is an abbreviation for Iterative Closest Point. In the following relational expression (2), i is a natural number. wi is a coefficient related to the difference between the i-th current predicted reflection intensity Ep(n) and the reflection intensity of the distance measurement point Pm. Also, wi is calculated by the object tracking device 20 using the current predicted reflection intensity Ep(n), the reflection intensity of the distance measurement point Pm, and a map. In the map for calculating wi, for example, wi becomes smaller as the absolute value of the difference between the current predicted reflection intensity Ep(n) and the reflection intensity of the distance measurement point Pm increases. Epi(n) is the i-th current predicted reflection intensity Ep(n). Ei is the reflection intensity of the i-th distance measurement point Pm. Here, m is the number of ranging points Pm extracted in step S102. Ppi(n) is the i-th current predicted ranging point Pp(n). Pmi is the i-th ranging point Pm. || || is the norm. c is a parameter for removing outliers, which are values ​​when the absolute value of the difference between Epi(n) and Ei is excessively large. Also, c is set by experiments, simulations, etc. so that outliers are removed.

[0065]

number

[0066] Therefore, the object tracking device 20 calculates a corrected predicted ranging point Pp_R(n) as shown in FIG. 10 using the calculated rotation matrix R(n) and translation matrix T(n) and the current predicted ranging point Pp(n).

[0067] Then, the object tracking device 20 associates the ranging point Pm with the corrected predicted ranging point Pp_R(n) so as to maximize the likelihood of the ranging point Pm and the corrected predicted ranging point Pp_R(n) by using a central projection method, a perspective projection method, a nearest neighbor search, feature matching, etc. In this way, the object tracking device 20 associates the ranging point Pm with the current predicted ranging point Pp(n), thereby associating the current observation information Io(n) with the current predicted information Ip(n).

[0068] 2, in step S112, the object tracking device 20 updates the current prediction information Ip(n) based on the corresponding ranging points Pm and predicted ranging points Pp. In this way, the object tracking device 20 tracks the tracked object 50.

[0069] For example, the object tracking device 20 uses the residual between the predicted ranging point Pp(n) and the ranging point Pm associated in step S110, an extended Kalman filter, a particle filter, a PHD filter, etc. In this way, the object tracking device 20 updates the current prediction information Ip(n) calculated in step S108. In this way, the object tracking device 20 calculates the current updated prediction information Ip_C(n) and tracks the tracked object 50.

[0070] Specifically, the object tracking device 20 calculates the current updated x-coordinate xp_C(n) of the center position Op from the residual, the filter, and the current predicted x-coordinate xp(n) of the center position Op. Also, the object tracking device 20 calculates the current updated y-coordinate yp_C(n) of the center position Op from the residual, the filter, and the current predicted y-coordinate yp(n) of the center position Op.

[0071] Furthermore, the object tracking device 20 calculates a current updated object length Lp_C(n) from the residual, the filter, and the current predicted object length Lp(n). Also, the object tracking device 20 calculates a current updated object width Wp_C(n) from the residual, the filter, and the current predicted object width Wp(n).

[0072] Furthermore, the object tracking device 20 calculates a current updated object speed Vp_C(n) from the residual, the filter, and the current predicted object speed Vp(n). The object tracking device 20 also calculates a current updated object yaw angle ψp_C(n) from the residual, the filter, and the current predicted object yaw angle ψp(n). The object tracking device 20 also calculates a current updated object yaw angular velocity ωp_C(n) from the residual, the filter, and the current predicted object yaw angular velocity ωp(n).

[0073] The object tracking device 20 also calculates a current update central distance rp_C(n) from the residual, the filter, and the current predicted central distance rp(n). The object tracking device 20 also calculates a current update relative angle θp_C(n) from the residual, the filter, and the current predicted relative angle θp(n). The object tracking device 20 also calculates a current update predicted ranging point Pp_C(n) from these.

[0074] Furthermore, the object tracking device 20 calculates a current update shape function rp_C(n)=f(θp_C(n)) from the calculated current update center distance rp_C(n) and current update relative angle θp_C(n). The current update shape function rp_C(n)=f(θp_C(n)) is expressed, for example, by a B-spline function. Note that the current update shape function rp_C(n)=f(θp_C(n)) is not limited to being expressed by a B-spline function, and may be a function indicating a quadrangle such as a rectangle, an elliptical function, a star-shaped convex polygon function, or the like.

[0075] Furthermore, the object tracking device 20 calculates a current updated reflection intensity Ep_C(n) and a current updated reflection function Ep_C(n)=g(θp_C(n)) from the current predicted reflection intensity Ep(n) as follows.

[0076] 11 and 12, when the incident angle αp is π / 2, the reflection intensity becomes very high. As a result, the reflection intensity when the incident angle αp is π / 2 becomes noise.

[0077] Therefore, the object tracking device 20 excludes the current predicted reflection intensity Ep(n) when the current predicted incident angle αp(n) is π / 2 from the current predicted reflection intensity Ep(n) calculated in step S108. Then, the object tracking device 20 creates a histogram from the remaining current predicted reflection intensity Ep(n) and the current updated relative angle θp_C(n) calculated above.

[0078] Specifically, the object tracking device 20 creates a histogram showing the relationship between the partitions for each currently updated relative angle θp_C(n) and the average predicted reflection intensity Ep_ave(n) for each partition, as shown in Fig. 13. Note that the average predicted reflection intensity Ep_ave(n) is calculated, for example, by taking a moving average, an arithmetic average, or a weighted average of the current predicted reflection intensity Ep(n) for the partitions for each currently updated relative angle θp_C(n).

[0079] Furthermore, noise such as variations in reflection intensity caused by light blocking, etc. occurs, and as a result, the reflection intensity contains a relatively large amount of noise.

[0080] Therefore, when the number of times when the calculated average predicted reflection intensity Ep_ave(n) is greater than the reflection threshold Ep_th is consecutive for a predetermined number of times or more, the object tracking device 20 sets the calculated average predicted reflection intensity Ep_ave(n) as the current updated reflection intensity Ep_C(n). Furthermore, when the number of times when the calculated average predicted reflection intensity Ep_ave(n) is greater than the reflection threshold Ep_th is not consecutive for a predetermined number of times or more, the object tracking device 20 keeps the current updated reflection intensity Ep_C(n) as the previous updated reflection intensity Ep_C(n-1). Furthermore, when the calculated average predicted reflection intensity Ep_ave(n) is smaller than the reflection threshold Ep_th, the object tracking device 20 keeps the previous updated reflection intensity Ep_C(n-1). The reflection threshold Ep_th and the predetermined number of times are set by experiment, simulation, or the like so that noise in the current updated reflection intensity Ep_C(n) is removed.

[0081] From the above, the object tracking device 20 calculates the currently updated x-coordinate xp_C(n) of the center position Op, the currently updated y-coordinate yp_C(n) of the center position Op, the currently updated object length Lp_C(n), and the currently updated object width Wp_C(n) as the currently updated predicted information Ip_C(n). Furthermore, the object tracking device 20 calculates the currently updated object speed Vp_C(n), the currently updated object yaw angle ψp_C(n), and the currently updated object yaw angular velocity ωp_C(n) as the currently updated predicted information Ip_C(n). Furthermore, the object tracking device 20 calculates the currently updated center distance rp_C(n), the currently updated relative angle θp_C(n), and the currently updated predicted ranging point Pp_C(n) as the currently updated predicted information Ip_C(n). Furthermore, the object tracking device 20 calculates a current update shape function rp_C(n)=f(θp_C(n)) as current update prediction information Ip_C(n). The object tracking device 20 also calculates a current update reflection intensity Ep_C(n) and a current update reflection function Ep_C(n)=g(θp_C(n)) as current update prediction information Ip_C(n).

[0082] 2, next, in step S114, the object tracking device 20 outputs the updated prediction information Ip_C calculated in step S112 as tracking information Ic to the vehicle ECU 30 via the in-vehicle LAN 18. As a result, the vehicle ECU 30 performs driving assistance for the vehicle 10, such as ACC, or automatic driving of the vehicle 10, based on, for example, the tracking information Ic acquired from the object tracking device 20, the vehicle attitude Ac acquired from the IMU 14, and the vehicle speed Vc acquired from the wheel speed sensor 16.

[0083] As described above, the object tracking device 20 performs object tracking. Next, how the object tracking device 20 suppresses a decrease in robustness in object tracking will be described.

[0084] Here, there is an extended object tracking technology called Extended Object Tracking, which recursively estimates the motion and shape states of a tracked object by using the sensor's ranging points obtained from the tracked object.In order to estimate the motion and shape states of a tracked object with high accuracy, it is important to obtain a correct update amount by accurately associating the observation information obtained from the tracked object with the information of the estimated state predicted from the previous time.

[0085] In response to this, an algorithm is known that uses reflection intensity information and geometric information to match point clouds, as described in Non-Patent Document 1. In the algorithm of Non-Patent Document 1, instantaneous reflection intensity information and geometric information are used, so the accuracy of matching point clouds decreases due to noise such as variations in reflection intensity information and geometric information and variations in reflection intensity caused by light blocking, etc. Therefore, the algorithm of Non-Patent Document 1 has low robustness in matching point clouds. For this reason, the algorithm of Non-Patent Document 1 cannot be simply used for object tracking.

[0086] In addition, in the case of matching based on geometric information, if the unevenness of the surface shape of the tracked object is extremely large or small, or if only a part of the vehicle is captured by the sensor, sufficient features cannot be obtained from the geometric information alone, and the accuracy of matching decreases. Note that a case where the unevenness of the surface shape of the tracked object is extremely large is, for example, a case where a number of needle-like objects are formed on the surface of the tracked object, such as an echinoderm such as a sea urchin. Furthermore, a case where the unevenness of the surface shape of the tracked object is extremely small is, for example, a case where the tracked object is formed with a smooth curve or plane, such as a vehicle.

[0087] For example, an object tracking device described in Japanese Patent No. 6895911 is known. This object tracking device uses a shape model that indicates the likelihood that each position on the tracked object will be a ranging point, and performs maximum likelihood estimation of the trajectory and shape model of the tracked object for each ranging point so that the sum of the probabilities that the ranging point position is on the tracked object is maximized. However, this object tracking device uses only information on the positions of the ranging points. Therefore, in a situation where the ranging points of the tracked object are only partially obtained, an error may occur in the correspondence in the point cloud of the observation points of the tracked object due to a change in the posture of the tracked object, as shown in FIG. 14. Therefore, in this object tracking device, the accuracy of the correspondence in the point cloud of the ranging points of the tracked object is reduced. Therefore, even in this object tracking device, the robustness of the correspondence in the point cloud is low.

[0088] Furthermore, a tracking device described in Japanese Patent No. 7211329 is known. In this tracking device, the state distribution of the target is predicted. In addition, an individual likelihood function that indicates the likelihood of the state distribution at each observation point of the target is corrected according to the number of observation points. Then, the corrected individual likelihood function is used to update the state distribution. However, in a situation where the observation points of the target can only be partially obtained, an error may occur in the correspondence in the point cloud of the observation points of the target due to a change in the posture of the target, as in the above case. Therefore, in the tracking device, the accuracy of the correspondence in the point cloud of the observation points of the target is reduced. Therefore, in this tracking device, the robustness of the correspondence in the point cloud is also low.

[0089] Furthermore, a tracking method described in Japanese Patent No. 7101535 is known. In this tracking method, in a first step, an initialization point cloud is obtained as a high-resolution three-dimensional point cloud obtained over a plurality of predefined time intervals. In a second step, a noisy reference model is generated from the initialization point cloud using a small number of measurement points. In a subsequent iterative step, an algorithm is used to estimate the pose of the target object at predefined time intervals to determine a temporal position cloud. Furthermore, a comparison is made with the reference model, and position information of the target object is calculated. Then, the reference model is continuously adapted to changes in the three-dimensional point cloud. However, in this tracking method, only information on the position of the target object is used. Therefore, as in the above, in a situation where the target object is only partially obtained, a change in the pose of the target object may cause an error in the correspondence in the point cloud of the target object, as shown in FIG. 15. Therefore, in this tracking method, the accuracy of the correspondence in the point cloud of the target object is reduced. Therefore, even in this tracking method, the robustness of the correspondence in the point cloud is low.

[0090] Furthermore, as described in the following Non-Patent Document 2, an algorithm is known that performs correspondence after correcting the deviation of the position or posture of the observed point cloud and the predicted object contour. However, in the algorithm described in Non-Patent Document 2, only information on the position of the object is used. Therefore, as in the above, in a situation where the object can only be partially obtained, an error may occur in the correspondence in the point cloud of the object due to a change in the posture of the object. Therefore, in the algorithm described in Non-Patent Document 2, the accuracy of the correspondence in the point cloud of the object decreases. Therefore, even in the algorithm described in Non-Patent Document 2, the robustness of the correspondence of the point cloud is low. Non-patent document 2 is K. -M. Dahlen, C. Lindberg, M. Yoneda and T. Ogawa, "An Improved B-spline Extended Object Tracking Model using the Iterative Closest Point Method," 2022 25th International Conference on Information Fusion (FUSION), Linkoping, Sweden, 2022, pp. 1-8, doi: 10.23919 / FUSION49751.2022.9841363.

[0091] In contrast to these, the object tracking device 20 of this embodiment serves as an acquisition unit that acquires position data and reflection data of the distance measurement point Pm detected by the distance measurement sensor 12 in step S100. Also, the object tracking device 20 serves as an observation unit that extracts the distance measurement point Pm that contributes to the value related to the shape of the tracked object 50 based on the position data and reflection data in step S102, and calculates the position and reflection intensity of the extracted distance measurement point Pm. Furthermore, the object tracking device 20 serves as a prediction unit that predicts the position and reflection intensity of the extracted distance measurement point Pm in step S108. Also, the object tracking device 20 serves as a correspondence calculation unit that associates the extracted distance measurement point Pm with the predicted distance measurement point Pp based on the position and reflection intensity of the extracted distance measurement point Pm and the position and reflection intensity of the predicted distance measurement point Pp in step S110. Furthermore, in step S112, the object tracking device 20 updates the position and reflection intensity of the predicted ranging point Pp based on the extracted ranging point Pm and the predicted ranging point Pp that correspond to each other, thereby serving as an update unit that tracks the tracked object 50. In addition, in step S108, the object tracking device 20 predicts the position and reflection intensity of the extracted ranging point Pm based on the position and reflection intensity of the updated predicted ranging point Pp.

[0092] By using the positions and reflection intensities of the extracted ranging points Pm and predicted ranging points Pp, the posture of the tracked object 50 is taken into consideration to associate the ranging points Pm with the predicted ranging points Pp, as shown in Fig. 16. In addition, since not only the instantaneous information but also the predicted information Ip is used, the accuracy of the point group association is prevented from decreasing due to noise such as variations in the positions and reflection intensities of the point groups, compared to the case where only the instantaneous information is used. Therefore, the robustness of the object tracking is prevented from decreasing. In Fig. 16, the ranging points Pm and the predicted ranging points Pp are indicated by white circles and dotted circles to indicate that the reflection intensities of the points are different.

[0093] Furthermore, the object tracking device 20 of the first embodiment also provides the following effects.

[0094] [1-1] In step S102, the object tracking device 20 calculates a convex hull for the distance measurement point Pm from the position data. Furthermore, the object tracking device 20 extracts the distance measurement point Pm that contributes to a value related to the shape of the tracked object 50 based on the calculated convex hull.

[0095] This makes it relatively easy to extract the distance measurement points Pm that contribute to values ​​related to the shape of the tracked object 50.

[0096] [1-2] In step S108, the object tracking device 20 predicts the reflection intensity for the predicted ranging point Pp based on the predicted distance Dp. This improves the prediction accuracy of the reflection intensity for the predicted ranging point Pp. As described above, the predicted distance Dp is the distance from the ranging sensor 12 to the predicted ranging point Pp.

[0097] [1-3] In step S108, the object tracking device 20 predicts the reflection intensity for the predicted ranging point Pp based on the incidence angle αp. This improves the prediction accuracy of the reflection intensity for the predicted ranging point Pp. As described above, the incidence angle αp is the angle between the line connecting the predicted ranging point Pp and the ranging sensor 12 and the tangent to the predicted ranging point Pp.

[0098] [1-4] In step S108, the object tracking device 20 predicts the reflection intensity for the predicted ranging point Pp based on the yaw angle of the tracked object 50, here the variance of the object yaw angle ψp. This improves the prediction accuracy of the reflection intensity for the predicted ranging point Pp.

[0099] [1-5] In step S110, the object tracking device 20 corrects the predicted ranging point Pp to approach the ranging point Pm extracted in step S102 based on the positions and reflection intensities of the extracted ranging point Pm and the predicted ranging point Pp. The object tracking device 20 also associates the corrected predicted ranging point Pp with the extracted ranging point Pm.

[0100] This makes it easier to associate the distance measurement points Pm extracted in step S102 with the predicted distance measurement points Pp.

[0101] [1-6] The value of the reflection intensity for the predicted ranging point Pp, here, the average predicted reflection intensity Ep_ave is assumed to be greater than the reflection threshold Ep_th. When this occurs a predetermined number of times in succession, the object tracking device 20 updates the reflection intensity for the predicted ranging point Pp in step S112.

[0102] As a result, reflection intensities with relatively high reliability remain, and reflection intensity values ​​that become noise are removed, improving the prediction accuracy of the predicted distance measurement point Pp in the next time.

[0103] [1-7] The object tracking device 20 updates the reflection intensity for the predicted ranging point Pp, except for the predicted ranging point Pp when the incident angle αp is π / 2.

[0104] This removes the reflection intensity that becomes noise, improving the prediction accuracy of the next predicted distance measurement point Pp.

[0105] (Variation 1) In the first embodiment, in step S102, the object tracking device 20 calculates the observation information Io(n) by extracting the distance measurement points Pm that contribute to the value related to the shape of the tracked object 50 using the convex hull for the distance measurement points Pm that belong to the group obtained by grouping. In contrast, the object tracking device 20 is not limited to extracting the distance measurement points Pm that contribute to the value related to the shape of the tracked object 50 using the convex hull.

[0106] For example, the object tracking device 20 may calculate the observation information Io(n) based on the distance from the ranging point Pm belonging to the group to the ranging sensor 12 and the angle of the ranging point Pm with respect to the ranging sensor 12.

[0107] Here, the distance from the ranging point Pm that contributes to the value related to the shape of the tracked object 50 to the ranging sensor 12 is smaller than the distance from the ranging point Pm that does not contribute to the value related to the shape of the tracked object 50 to the ranging sensor 12. For example, as shown in Fig. 17, the distance from the ranging point Pm on the outer periphery to the ranging sensor 12 when the preceding vehicle is viewed from above is smaller than the distance from the ranging point Pm inside the preceding vehicle to the ranging sensor 12 when the preceding vehicle is viewed from above.

[0108] Therefore, the object tracking device 20 calculates each distance from the ranging points Pm belonging to the group to the ranging sensor 12, and each angle of the ranging points Pm with respect to the ranging sensor 12. The object tracking device 20 also extracts ranging points Pm whose calculated distances are equal to or less than the distance threshold in a predetermined angle section Δθ. As a result, the object tracking device 20 removes ranging points Pm that do not contribute to values ​​related to the shape of the tracked object 50, here, ranging points Pm inside the preceding vehicle when the preceding vehicle is viewed from above. The distance threshold and the angle section Δθ are set by experiments, simulations, etc. so that ranging points Pm that contribute to values ​​related to the shape of the tracked object 50 are extracted.

[0109] As described above, even if the object tracking device 20 extracts the distance measurement points Pm that contribute to values ​​related to the shape of the tracked object 50, the same effects as those of the first embodiment can be achieved.

[0110] (Variation 2) In the first embodiment, the object tracking device 20 calculates the observation information Io, the prediction information Ip, and the updated prediction information Ip_C, and associates the observation information Io with the prediction information Ip, for one wavelength of the exploration wave from the distance measuring sensor 12. In contrast, the object tracking device 20 may calculate the observation information Io, the prediction information Ip, and the updated prediction information Ip_C, and associate the observation information Io with the prediction information Ip, for each of multiple wavelengths.

[0111] For example, the object tracking device 20 may process each wavelength of radio waves such as millimeter waves, each wavelength of light of LiDAR, and each wavelength of ultrasonic waves as the wavelength of the search wave. Also, the object tracking device 20 may further include a camera, and the pixel position of the camera may be linked to the position of the ranging point Pm, and the object tracking device 20 may process each of the RGB of the camera.

[0112] As described above, even if the processing of the object tracking device 20 is performed for each of a plurality of wavelengths, the same effect as that of the first embodiment is achieved. Furthermore, by performing the processing of the object tracking device 20 for each of a plurality of wavelengths, the accuracy of the association between the observation information Io and the prediction information Ip is increased. Therefore, the accuracy of the association between the observation information Io and the prediction information Ip is improved.

[0113] Second embodiment In the second embodiment, the vehicle 10 further includes an image sensor 40, as shown in Fig. 18. Also, the processing of the object tracking device 20 differs from that of the first embodiment. Other than this, the second embodiment is similar to the first embodiment.

[0114] The image sensor 40 is a camera or the like, and captures an image of the tracked object 50 to detect a point cloud consisting of at least one ranging point Pm, which is a point for each position on the tracked object 50. Furthermore, the image sensor 40 generates position data and brightness data of the detected ranging point Pm. The image sensor 40 also outputs the generated position data and brightness data to the object tracking device 20 via the in-vehicle LAN 18. Note that the brightness data is data related to the brightness of the ranging point Pm in the image captured by the image sensor 40.

[0115] The object tracking device 20 uses brightness data instead of reflection data to track the tracked object 50. Therefore, in the second embodiment, the reflection data in the first embodiment is replaced with brightness data. Furthermore, the reflection intensity in the first embodiment is replaced with brightness. Furthermore, the reflection function in the first embodiment is replaced with a brightness function. The brightness function is a function that represents the relationship between the relative angle θp and brightness.

[0116] The object tracking device 20 of the second embodiment is configured as described above. The second embodiment also provides the same effects as the first embodiment.

[0117] Third embodiment In the third embodiment, the form of the distance measuring sensor 12 is different from that of the first embodiment. Also, the process of the object tracking device 20 is different from that of the first embodiment. Other than these, the third embodiment is similar to the first embodiment.

[0118] The distance measurement sensor 12 detects the Doppler velocity of the distance measurement point Pm, for example, by using the frequency of the search wave. Furthermore, the distance measurement sensor 12 generates velocity data in addition to the position data. The distance measurement sensor 12 outputs the position data and velocity data to the object tracking device 20 via the in-vehicle LAN 18. Note that the Doppler velocity here is the relative velocity of the distance measurement point Pm with respect to the distance measurement sensor 12. Furthermore, the velocity data is data related to the Doppler velocity of the distance measurement point Pm.

[0119] The object tracking device 20 performs the above-mentioned tracking of the tracked object 50 using velocity data instead of reflection data. Therefore, in the third embodiment, the reflection data in the first embodiment is replaced with velocity data. Furthermore, the reflection intensity in the first embodiment is replaced with Doppler velocity. Furthermore, the reflection function in the first embodiment is replaced with a Doppler velocity function. The Doppler velocity function is a function that represents the relationship between the relative angle θp and the Doppler velocity.

[0120] The object tracking device 20 of the third embodiment is configured as described above. The third embodiment also provides the same effects as the first embodiment.

[0121] (Other embodiments) The present disclosure is not limited to the above-described embodiments, and appropriate modifications can be made to the above-described embodiments. Furthermore, in each of the above-described embodiments, it goes without saying that the elements constituting the embodiments are not necessarily essential, except in cases where they are particularly clearly stated as essential or where they are clearly considered essential in principle.

[0122] The acquisition unit, observation unit, prediction unit, correspondence calculation unit, update unit, and the method thereof described in the present disclosure may be realized by a dedicated computer provided by configuring a processor and memory programmed to execute one or more functions embodied in a computer program. Alternatively, the acquisition unit, observation unit, prediction unit, correspondence calculation unit, update unit, and the method thereof described in the present disclosure may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the acquisition unit, observation unit, prediction unit, correspondence calculation unit, update unit, and the method thereof described in the present disclosure may be realized by one or more dedicated computers configured by a combination of a processor and memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transient tangible recording medium as instructions executed by a computer.

[0123] In each of the above embodiments, the prediction information Ip(n) is calculated by using a constant velocity and constant acceleration motion model, but the present invention is not limited to using a constant velocity and constant acceleration motion model. For example, the prediction information Ip(n) may be calculated by a method using multiple models such as the IMM method. Note that IMM is an abbreviation for Interacting Multiple Model.

[0124] In each of the above embodiments, the current predicted reflection intensity Ep(n) is calculated using the current predicted distance Dp(n), k, h(αp(n)), current predicted variance σψ(n), and the previous predicted distance Dp(n-1) and h(αp(n-1)). In contrast, when calculating the current predicted reflection intensity Ep(n), it is not necessary to use h(αp(n)) and h(αp(n-1)), which are values ​​related to the angle of incidence αp. The current predicted reflection intensity Ep(n) may be calculated using the current predicted distance Dp(n), k, current predicted variance σψ(n), and the previous predicted distance Dp(n-1).

[0125] Furthermore, the current predicted reflection intensity Ep(n) may be smoothed using a Gaussian filter or the like. For example, as shown in the following relational expression (3), the current predicted reflection intensity Ep(n) is smoothed by multiplying it by the following G(n). In the following relational expression (3), Ep_S(n) is the smoothed current predicted reflection intensity Ep(n). σθ is the variance related to the current predicted relative angle θp(n).

[0126]

number

[0127] In each of the above embodiments, the object tracking device 20 is used in the vehicle 10, but is not limited to this. For example, the object tracking device 20 may be used in a traffic monitoring system or the like.

[0128] In the first embodiment, the reflection function Ep=g(θp) is expressed by a histogram, but is not limited to being expressed by a histogram. The reflection function Ep=g(θp) may be an approximate function of the reflection intensity with respect to the relative angle θp.

[0129] In the first embodiment, the reflection intensity is used when tracking an object. However, instead of the reflection intensity, a reflectance, which is the ratio of the reflection intensity to the incident intensity of the search wave, may be used.

[0130] The above-described embodiments and modifications may be combined as appropriate. [Explanation of symbols]

[0131] 10 Vehicles 12 Distance sensor 14 IMU 16 Wheel speed sensor 20 Object Tracking Device 30 Vehicle ECU 40 Image Sensor 50 Tracked Objects

Claims

1. An object tracking device, comprising: an acquisition unit (S100) for acquiring position data of a distance measurement point (Pm) detected by a distance measurement sensor (12) and reflection data which is data related to the reflection intensity of a search wave of the distance measurement sensor reflected at the distance measurement point; an observation unit (S102) for extracting the distance measurement points that contribute to a value related to the shape of the tracked object (50) based on the position data and the reflection data, and calculating values ​​related to the positions and reflection intensity of the extracted distance measurement points; a prediction unit (S108) for predicting values ​​related to positions and reflection intensities of the distance measurement points extracted by the observation unit; a correspondence calculation unit (S110) that performs correspondence between the distance measurement points extracted by the observation unit and the predicted distance measurement points based on values ​​relating to positions and reflection intensity of the distance measurement points extracted by the observation unit and values ​​relating to positions and reflection intensity of predicted distance measurement points (Pp) that are the distance measurement points predicted by the prediction unit; an update unit (S112) that tracks the tracked object by updating values ​​related to positions and reflection intensities of the predicted ranging points based on the ranging points and the predicted ranging points extracted by the observation units corresponding to each other; Equipped with The prediction unit predicts values ​​related to the position and reflection intensity of the predicted ranging point extracted by the observation unit based on values ​​related to the position and reflection intensity of the predicted ranging point updated by the update unit.

2. The object tracking device according to claim 1 , wherein the observation unit calculates a convex hull for the distance measurement points from the position data, and extracts the distance measurement points that contribute to a value related to a shape of the tracked object based on the calculated convex hull.

3. The object tracking device according to claim 1 , wherein the prediction unit predicts a value relating to reflection intensity for the predicted ranging point based on a distance (Dp) from the distance measuring sensor to the predicted ranging point.

4. The object tracking device of claim 1 or 2, wherein the prediction unit predicts a value related to reflection intensity for the predicted ranging point based on an angle (αp) between a line connecting the ranging sensor and the predicted ranging point and a tangent to the predicted ranging point.

5. The object tracking device according to claim 1 or 2, wherein the prediction unit predicts a value relating to reflection intensity for the predicted distance measurement point based on a yaw angle (ψp) relating to the tracked object.

6. The correspondence calculation unit is performing a correction so as to bring the predicted ranging point closer to the ranging point extracted by the observation unit, based on values ​​relating to the position and reflection intensity of the ranging point extracted by the observation unit and values ​​relating to the position and reflection intensity of the predicted ranging point; 3. The object tracking device according to claim 1, further comprising: a step of associating the corrected predicted distance measurement points with the distance measurement points extracted by the observation unit.

7. The object tracking device according to claim 1 or 2, wherein the update unit updates the value regarding the reflection intensity for the predicted ranging point when the number of times that the value regarding the reflection intensity for the predicted ranging point is greater than the threshold value is equal to or greater than a predetermined number of consecutive times.

8. The object tracking device of claim 1 or 2, wherein the update unit updates values ​​regarding the position and reflection intensity of the predicted ranging point, except for the predicted ranging point when the angle (αp) between the line connecting the ranging sensor and the predicted ranging point and the tangent to the predicted ranging point is π / 2.

9. the distance measurement sensor detects the distance measurement point using a plurality of wavelengths; the acquiring unit acquires the position data and the reflection data for each of the plurality of wavelengths, the observation unit extracts the distance measurement points that contribute to a value related to a shape of the tracked object for each of the plurality of wavelengths, and calculates values ​​related to positions and reflection intensities of the extracted distance measurement points; the prediction unit predicts values ​​related to positions and reflection intensities of the distance measurement points extracted by the observation unit for each of the plurality of wavelengths; the correspondence calculation unit performs correspondence between the ranging points extracted by the observation unit and the predicted ranging points for each of the plurality of wavelengths; The object tracking device according to claim 1 , wherein the update unit updates values ​​relating to a position and a reflection intensity of the predicted distance measurement point for each of the plurality of wavelengths.

10. An object tracking device, comprising: an acquisition unit (S100) that acquires position data of a distance measurement point (Pm) detected by an image sensor (40) and luminance data that is data related to the luminance of the distance measurement point in an image captured by the image sensor; an observation unit (S102) for extracting the distance measurement points that contribute to a value related to the shape of the tracked object (50) based on the position data and the brightness data, and calculating values ​​related to the positions and brightness of the extracted distance measurement points; a prediction unit (S108) for predicting values ​​related to positions and luminance of the distance measurement points extracted by the observation unit; a correspondence calculation unit (S110) that performs correspondence between the distance measurement points extracted by the observation unit and the predicted distance measurement points based on values ​​related to positions and luminance of the distance measurement points extracted by the observation unit and values ​​related to positions and luminance of predicted distance measurement points (Pp) that are the distance measurement points predicted by the prediction unit; an update unit (S112) that tracks the tracked object by updating values ​​related to positions and brightness of the predicted ranging points based on the ranging points and the predicted ranging points extracted by the observation units corresponding to each other; Equipped with The prediction unit predicts values ​​related to position and brightness for the predicted ranging point extracted by the observation unit based on values ​​related to position and brightness for the predicted ranging point updated by the update unit.

11. An object tracking device, comprising: an acquisition unit (S100) for acquiring position data of a distance measurement point (Pm) detected by a distance measurement sensor (12) and velocity data which is data related to a Doppler velocity for the distance measurement point; an observation unit (S102) for extracting the distance measurement points that contribute to a value related to the shape of the tracked object (50) based on the position data and the velocity data, and calculating values ​​related to the position and Doppler velocity of the extracted distance measurement points; a prediction unit (S108) for predicting values ​​related to positions and Doppler velocities of the distance measurement points extracted by the observation unit; a correspondence calculation unit (S110) that performs correspondence between the distance measurement points extracted by the observation unit and the predicted distance measurement points based on values ​​relating to positions and Doppler speeds of the distance measurement points extracted by the observation unit and values ​​relating to positions and Doppler speeds of predicted distance measurement points (Pp) that are the distance measurement points predicted by the prediction unit; an update unit (S112) that tracks the tracked object by updating values ​​related to positions and Doppler velocities of the predicted ranging points based on the ranging points and the predicted ranging points extracted by the observation units corresponding to each other; Equipped with An object tracking device in which the prediction unit predicts values ​​related to the position and Doppler velocity for the predicted ranging point extracted by the observation unit based on values ​​related to the position and Doppler velocity for the predicted ranging point updated by the update unit.

12. An object tracking program, comprising: The object tracking device, an acquisition unit (S100) for acquiring position data of a distance measurement point (Pm) detected by a distance measurement sensor (12) and reflection data which is data related to the reflection intensity of a search wave of the distance measurement sensor reflected at the distance measurement point; an observation unit (S102) for extracting the distance measurement points that contribute to a value related to the shape of the tracked object (50) based on the position data and the reflection data, and calculating values ​​related to the positions and reflection intensity of the extracted distance measurement points; a prediction unit (S108) for predicting values ​​related to positions and reflection intensities of the distance measurement points extracted by the observation unit; a correspondence calculation unit (S110) that performs correspondence between the distance measurement points extracted by the observation unit and the predicted distance measurement points based on values ​​relating to positions and reflection intensity of the distance measurement points extracted by the observation unit and values ​​relating to positions and reflection intensity of predicted distance measurement points (Pp) that are the distance measurement points predicted by the prediction unit; and updating values ​​relating to positions and reflection intensities of the predicted ranging points based on the ranging points and the predicted ranging points extracted by the observation units corresponding to each other, thereby functioning as an update unit (S112) that tracks the tracked object; The prediction unit is an object tracking program that predicts values ​​related to the position and reflection intensity of the predicted ranging point extracted by the observation unit based on values ​​related to the position and reflection intensity of the predicted ranging point updated by the update unit.

13. An object tracking program, comprising: The object tracking device, an acquisition unit (S100) for acquiring position data of a distance measurement point (Pm) detected by an image sensor (40) and brightness data which is data related to the brightness of the distance measurement point in an image captured by the image sensor; an observation unit (S102) for extracting the distance measurement points that contribute to a value related to the shape of the tracked object (50) based on the position data and the brightness data, and calculating values ​​related to the positions and brightness of the extracted distance measurement points; a prediction unit (S108) for predicting values ​​related to positions and luminances of the distance measurement points extracted by the observation unit; a correspondence calculation unit (S110) that performs correspondence between the distance measurement points extracted by the observation unit and the predicted distance measurement points based on values ​​related to positions and luminance of the distance measurement points extracted by the observation unit and values ​​related to positions and luminance of predicted distance measurement points (Pp) that are the distance measurement points predicted by the prediction unit; and updating values ​​relating to positions and luminance of the predicted ranging points based on the ranging points and the predicted ranging points extracted by the observation units corresponding to each other, thereby functioning as an update unit (S112) that tracks the tracked object; The prediction unit predicts values ​​related to position and brightness of the predicted ranging point extracted by the observation unit based on values ​​related to position and brightness of the predicted ranging point updated by the update unit.

14. An object tracking program, comprising: The object tracking device, an acquisition unit (S100) for acquiring position data of a distance measurement point (Pm) detected by a distance measurement sensor (12) and velocity data which is data related to the Doppler velocity of the distance measurement point; an observation unit (S102) for extracting the distance measurement points that contribute to values ​​relating to the shape of the tracked object (50) based on the position data and the velocity data, and calculating values ​​relating to the positions and Doppler velocities of the extracted distance measurement points; a prediction unit (S108) for predicting values ​​related to the positions and Doppler velocities of the range points extracted by the observation unit; a correspondence calculation unit (S110) that performs correspondence between the distance measurement points extracted by the observation unit and the predicted distance measurement points based on values ​​relating to positions and Doppler speeds of the distance measurement points extracted by the observation unit and values ​​relating to positions and Doppler speeds of predicted distance measurement points (Pp) that are the distance measurement points predicted by the prediction unit; and updating values ​​relating to positions and Doppler velocities of the predicted ranging points based on the ranging points and the predicted ranging points extracted by the observation units corresponding to each other, thereby functioning as an update unit (S112) that tracks the tracked object; An object tracking program in which the prediction unit predicts values ​​related to the position and Doppler velocity for the predicted ranging point extracted by the observation unit based on values ​​related to the position and Doppler velocity for the predicted ranging point updated by the update unit.

Images