Target tracking device and target tracking method

Abstract

The system includes: an observation value processing unit (110) that acquires the direction cosine for each observed value of a surrounding moving object detected based on the detection data of multiple sensors; a correlation processing unit (120) that correlates the observed values ​​based on the direction cosine; a first integration unit (140) that integrates the observed values ​​correlated by the correlation processing unit; an identity determination unit (150) that determines whether the difference between the direction cosine of the observed value and the direction cosine of the stored track is within the range of the residual covariance matrix; and a second integration unit (160) that integrates the observed value and the track and outputs it if the identity determination unit determines that it is within the range of the residual covariance matrix.

Description

Technical Field

[0001] The disclosed technology relates to a target tracking technology that integrates detection data from multiple sensors and is used for target tracking.

Background Art

[0002] As a method for integrating detection data obtained from multiple sensors, there is a technique called sensor fusion. In sensor fusion, the association of detection data of the same detection object is determined from all the detection data obtained by each sensor.

[0003] Patent Document 1 discloses a "road marking recognition device" that detects the same detection object using detection data obtained from two sensors. The "road marking recognition device" described in Patent Document 1 irradiates a radar (equivalent to a sensor) when characteristic points (white line road markings) continuously exceeding a predetermined length are not extracted in the forward lane area captured by a camera (equivalent to a sensor), and enables the detection of road markings represented by reflectors intermittently laid on the road, so that the road markings other than the white line road markings can be recognized.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, if sensor fusion is used to perform target tracking of a moving object such as a surrounding vehicle, when attempting to associate the detection data of the same detection object, there may be a displacement in the observed value after association, resulting in a separate track, and there is a problem that the accuracy of target tracking becomes low. The "road lane marking recognition device" described in Patent Document 1 recognizes road lane markings and is not configured to detect moving objects in the first place, and therefore cannot solve the above problem.

[0006] This disclosure aims to solve the above-mentioned problems and to achieve high-precision target tracking using sensor fusion. [Means for solving the problem]

[0007] The target tracking device of this disclosure is Based on the detection data from multiple sensors, each of the observed values ​​of the surrounding moving objects In relation to the direction of movement of the surrounding moving object An observation processing unit that obtains direction cosines, A correlation processing unit that calculates the correlation between observed values ​​based on the direction cosine, A first integration unit integrates the observed values ​​that have been correlated by the correlation processing unit, An identity determination unit that determines whether the difference between the direction cosine of the observed value and the direction cosine of the stored track is within the range of the residual covariance matrix, A second integration unit that integrates the observed value and the track and outputs the result when the identity determination unit determines that the result is within the range of the residual covariance matrix, It is something that is provided. [Effects of the Invention]

[0008] According to this disclosure, it is possible to achieve highly accurate target tracking using sensor fusion. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of a first configuration of the target tracking device of this disclosure. [Figure 2] Figure 2 shows an example of a second configuration of the target tracking device of this disclosure. [Figure 3] Figure 3 shows an example of a configuration when the target tracking device of this disclosure is applied to a target tracking system. [Figure 4] Figure 4 is a flowchart showing an example of the process when the target tracking device is as shown in Figure 1. [Figure 5] Figure 5 is a flowchart showing an example of the process when the target tracking device is as shown in Figure 2. [Figure 6] Figure 6 is a flowchart showing an example of the processing according to Embodiment 1 when the target tracking device is as shown in Figure 3. [Figure 7] Figure 7 is a flowchart showing an example of the processing according to Embodiment 2 in the case of the target tracking device shown in Figure 3. [Figure 8] Figure 8 is a flowchart showing an example of the processing according to Embodiment 3 in the case of the target tracking device shown in Figure 3. [Figure 9] Figure 9 shows a first example of a hardware configuration for realizing the functions of the configuration of this disclosure. [Figure 10] Figure 10 shows a second example of a hardware configuration for realizing the functionality of the configuration described herein. [Modes for carrying out the invention]

[0010] To further illustrate this disclosure, embodiments of this disclosure will be described below with reference to the accompanying drawings.

[0011] Embodiment 1. Embodiment 1 describes the basic configuration example of this disclosure and the configuration example related to Embodiment 1.

[0012] A first configuration example of a target tracking device according to Embodiment 1 of this disclosure will be described. Figure 1 shows an example of a first configuration of the target tracking device of this disclosure. The target tracking device 100 integrates observational values ​​based on detection data from multiple sensors and uses them to track the target. The target tracking device 100 shown in Figure 1 includes an observation value processing unit 110, a correlation processing unit 120, and a tracking filter unit 130.

[0013] The observation processing unit 110 acquires detection data from multiple sensors and processes the observed value for each detection data. The observation processing unit 110 obtains the direction cosine for each observed value of the surrounding moving object detected based on the detection data from each of the multiple sensors. Specifically, the observation processing unit 110 in this embodiment acquires the velocity vector and direction cosine for each observed value of a surrounding moving object detected based on the detection data of multiple sensors. The method for acquiring the velocity vector and direction cosine will be described later.

[0014] The correlation processing unit 120 calculates the correlation between the observed values. The correlation processing unit 120 calculates the correlation between the observed values ​​based on the direction cosines. Specifically, the correlation processing unit 120 according to this embodiment calculates the correlation between the observed values ​​based on the velocity vector and the direction cosine.

[0015] The tracking filter unit 130 integrates the observed values ​​when correlated observed values ​​are input, and integrates the observed values ​​and the stored track if the integrated observed values ​​and the stored track are of the same object. The tracking filter unit 130 shown in Figure 1 is composed of an integration unit (first integration unit) 140, an identity determination unit 150, and a track integration unit (second integration unit) 160.

[0016] The integration unit (first integration unit) 140 of the tracking filter unit 130 integrates the correlated observed values. Specifically, the integration unit (first integration unit) 140 integrates the observed values ​​that have been correlated by the correlation processing unit 120. The integration unit (first integration unit) 140 integrates the observed values, for example, by calculating the smoothed values ​​of the correlated observed values. The method for calculating the smoothed values ​​will be described later.

[0017] The identity determination unit 150 of the tracking filter unit 130 determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object). Specifically, the identity determination unit 150 determines whether the difference between the direction cosine of the observed value and the direction cosine of the stored track is within the range of the residual covariance matrix. More specifically, the identity determination unit 150 according to this embodiment determines whether the difference between the velocity vector of the observed value and the velocity vector of the stored track, and the difference between the direction cosine of the observed value and the direction cosine of the stored track, are within the range of the residual covariance matrix.

[0018] The track integration unit (second integration unit) 160 of the tracking filter unit 130 integrates the observed values ​​and track for the same object (surrounding moving object). Specifically, the track integration unit (second integration unit) 160 integrates the observed values ​​and the track and outputs the result when the identity determination unit determines that the values ​​are within the range of the residual covariance matrix. The track integration unit (second integration unit) 160 integrates the observed values ​​and the track using an extended Kalman filter process, with the direction cosines of the observed values ​​correlated by the correlation processing unit. Alternatively, the track integration unit (second integration unit) 160 integrates the observed values ​​and the track using the smoothed values. Alternatively, the track integration unit (second integration unit) 160 integrates the observed values ​​and the track using a linear Kalman filter process, using the positions of surrounding moving objects included in the correlated observed values.

[0019] In addition to the above configuration, the target tracking device 100 may also include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown). A control unit (not shown) controls the entire target tracking device 100 and its individual components. For example, the control unit activates the target tracking device 100 according to an external command. The control unit also controls the state of the target tracking device 100 (operating state = state such as start, shutdown, sleep, etc.). A storage unit (not shown) stores the data used by the target tracking device 100. For example, the storage unit stores the output (output data) from each component of the target tracking device 100 and outputs the requested data to the requesting component. The communication unit (not shown) communicates with external devices. For example, it communicates between the target tracking device 100 (100A) and peripheral devices (e.g., mobile device). For example, if the target tracking device 100 and the mobile device are not connected by a wire, the communication unit (not shown) has the function of communicating between the target tracking device 100 and the mobile device. The communication unit (not shown) also has the function of communicating with an external device, such as a server device. The control unit (not shown), storage unit (not shown), and communication unit (not shown) are the same in the embodiments described later.

[0020] A second configuration example of the target tracking device according to Embodiment 1 of this disclosure will be described. Figure 2 shows an example of a second configuration of the target tracking device of this disclosure. The target tracking device 100 also has a function to correct the position included in the observed values ​​before they are integrated with the track. The target tracking device 100 shown in Figure 2 includes an observation value processing unit 110, a correlation processing unit 120, a tracking filter unit 130, and a position correction unit 170. The observation processing unit 110 is configured in the same way as the observation processing unit 110 described earlier. The correlation processing unit 120 is configured in the same way as the correlation processing unit 120 described earlier.

[0021] The tracking filter unit 130 integrates the observed values ​​when correlated observed values ​​are input, and integrates the observed values ​​and the stored track if the integrated observed values ​​and the stored track are of the same object. The tracking filter unit 130 shown in Figure 2 is composed of an integration unit (first integration unit) 140, an identity determination unit 150, and a track integration unit (second integration unit) 160. The integration unit (first integration unit) 140 is configured in the same manner as the integration unit (first integration unit) 140 described earlier.

[0022] The identity determination unit 150 determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object). The identity determination unit 150 further outputs the correlated observed values ​​to the position correction unit 170 when it corrects the position included in the observed values ​​if the correlated observed values ​​and the track stored in the track management unit 180 belong to the same object (surrounding moving object).

[0023] The position correction unit 170 corrects the position indicated by the correlated observed values. The position correction unit 170 corrects the position included in the observed values ​​before integration with the track. Specifically, the position correction unit 170 corrects the position included in the observed values ​​before integration with the track when the correlated observed values ​​are the same moving object as the moving object in the track stored in the track management unit 180. For example, the position correction unit 170 replaces the camera observation position with the millimeter-wave observation position and outputs it to the track integration unit (second integration unit) 160 of the tracking filter unit 130. Specifically, the position correction unit 170 replaces the camera observation position, which is an observed value based on the detection data of the imaging device, with the millimeter-wave observation position, which is an observed value based on the detection data of the millimeter-wave radar device.

[0024] The track integration unit (second integration unit) 160 can integrate the track with the observed values ​​corrected by the position correction unit 170. The track integration unit (second integration unit) 160 integrates the track with the observed values ​​using observed values ​​in which the position (camera observation position), which is an observed value based on the detection data of the imaging device, has been replaced with the position (millimeter-wave observation position), which is an observed value based on the detection data of the millimeter-wave radar device. The track integration unit (second integration unit) 160 integrates the track with the observed values ​​by linear Kalman filtering using the positions of surrounding moving objects included in the correlated observed values. The track integration unit (second integration unit) 160 may be configured to have the above-described functions in addition to the functions of the track integration unit (second integration unit) 160 already described. In this case, the track integration unit (second integration unit) 160 may be configured to, for example, determine whether to perform position correction.

[0025] Next, we will describe an example of a system configuration that includes a target tracking device. Figure 3 shows an example of a configuration when the target tracking device of this disclosure is applied to a target tracking system. The target tracking system 1 includes the target tracking device 100 according to Embodiment 1. Figure 3 shows the target tracking system 1 including the configuration of the target tracking device 100 shown in Figure 2. Embodiment 1 will be described assuming that the sensor consists only of a camera, or that the sensor consists of a sensor with low positional accuracy. In the following description, the target tracking system 1 and target tracking device 100 according to Embodiment 1 will be referred to as target tracking system 1(1A) and target tracking device 100(100A) to distinguish them from target tracking system 1 and target tracking device 100 according to other embodiments. The target tracking system 1(1A) shown in Figure 3 comprises a first sensor 11, a second sensor 12, a target tracking device 100(100A), and an output destination device 300.

[0026] The first sensor 11 and the second sensor 12 are sensors included in the plurality of sensors in this disclosure. The first sensor 11 and the second sensor 12 are sensors that receive light or electromagnetic waves emitted or reflected from an object, apply signal processing or image processing, and measure positional information of surrounding objects, such as the position, velocity, and azimuth angle of the object. Examples include millimeter-wave radar, laser radar, ultrasonic sensors, infrared sensors, and optical cameras. Surrounding objects are, for example, moving objects (surrounding moving objects) that are likely to exist around a moving object on which a sensor is mounted, with respect to the moving object on which the sensor is mounted. In the following explanation, we assume that the first sensor 11 and the second sensor 12 each consist only of cameras. However, the first sensor 11 and the second sensor 12 may each be composed of sensors with low positional accuracy.

[0027] The positions of the first sensor 11 and the second sensor 12, as well as the range in which they can detect objects, are assumed to be known. The mounting positions of the first sensor 11 and the second sensor 12 can be set arbitrarily. The target tracking device 100 (100A) can improve recognition accuracy by integrating the observation data of the same object detected by the first sensor 11 and the second sensor 12 into a single piece of information. Therefore, it is desirable that the detection ranges for objects of the first sensor 11 and the second sensor 12 overlap (have a common area).

[0028] The position information of the object measured by the first sensor 11 is used as the detection data for the first sensor 11. On the other hand, the position information of the object measured by the second sensor 12 is used as the detection data for the second sensor 12.

[0029] The target tracking device 100 (100A) integrates observational values ​​based on detection data from multiple sensors and uses them to track the target. The target tracking device 100 (100A) is implemented, for example, by a CPU that executes a program stored in memory and processing circuits such as a system LSI, as will be described later. The target tracking device 100 (100A) shown in Figure 3 includes an observation value processing unit 110, a correlation processing unit 120, a tracking filter unit 130, a position correction unit 170, a track management unit 180, and a track output unit 190.

[0030] The observation processing unit 110 acquires the velocity vector and direction cosine for each observed value of the surrounding moving object detected based on the detection data from multiple sensors. Specifically, in this embodiment, when the target tracking device 100 (100A) receives detection data from the first sensor 11 and detection data from the second sensor 12, the observation data processing unit 110 receives the detection data from the first sensor and the detection data from the second sensor, processes each detection data as observation data as necessary, and outputs it to the correlation processing unit 120 at predetermined processing cycles.

[0031] The correlation processing unit 120 calculates the correlation between the observed values ​​based on the velocity vector and the direction cosine. The correlation processing unit 120 in this embodiment determines the correspondence between the observation data and the object's track data from the track management unit 180, which will be described later. Furthermore, it outputs the observation data, the object's track data, and the assignment data summarizing this correspondence to the integration unit (first integration unit) 140 of the tracking filter unit 130.

[0032] The tracking filter unit 130 integrates the observed values ​​when correlated observed values ​​are input, and integrates the observed values ​​and the stored track if the integrated observed values ​​and the stored track are of the same object. The tracking filter unit 130 shown in Figure 3 is composed of an integration unit (first integration unit) 140, an identity determination unit 150, and a track integration unit (second integration unit) 160.

[0033] The integration unit (first integration unit) 140 of the tracking filter unit 130 integrates the observed values ​​that have been correlated by the correlation processing unit 120. The integration unit (first integration unit) 140 integrates the observed values ​​by, for example, calculating the smoothed value of the correlated observed values. The integration unit (first integration unit) 140 outputs the smoothed value obtained by integrating the observed values ​​to the identity determination unit 150.

[0034] The identity determination unit 150 of the tracking filter unit 130 determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object). The identity determination unit 150 according to this embodiment outputs to the track integration unit (second integration unit) 160 observation data (observed values) that have a corresponding relationship to the same object (surrounding moving object), or observation data that is highly likely to have a corresponding relationship. When correcting the position of correlated observed values, the identity determination unit 150 outputs to the position correction unit 170 the observed data (observed values) that have a corresponding relationship to the same object (surrounding moving object), or observed data (observed values) that have been associated with a high probability of a corresponding relationship.

[0035] The track integration unit (second integration unit) 160 of the tracking filter unit 130 integrates the observed values ​​and track for the same object (surrounding moving object). The track integration unit (second integration unit) 160 according to this embodiment updates object data based on correlation data and outputs it to the track management unit 180. The object data includes, for example, the object's state vector such as the position, velocity, and acceleration of the detected object; the number of detections of the observation data (observed values) of the first sensor 11 and the second sensor 12 used to generate and update this state vector; the observation data corresponding to the most recent (the most recent can also be expressed as the most recent from the current time) of the observation data (observed values) of the first sensor 11 and the second sensor 12 (hereinafter referred to as the latest correlation data); the elapsed time since the latest observation data (observed values) of the first sensor 11 and the second sensor 12 were detected; the number of lost data points, etc. Note that multiple objects are detected by the first sensor 11 and the second sensor 12, and object data is created for each object.

[0036] The state vector of the object data is updated using known methods such as the least squares method, Kalman filter, or particle filter.

[0037] The position correction unit 170 corrects the position included in the observed values ​​before they are integrated with the track. Specifically, the position correction unit 170 corrects the position included in the observed values ​​before they are integrated with the track when the correlated observed values ​​are the same moving object as the moving object in the track stored in the track management unit 180. For example, the position correction unit 170 replaces the camera's observation position with the millimeter-wave observation position and outputs it to the track integration unit (second integration unit) 160 of the tracking filter unit 130.

[0038] The track management unit 180 manages the track. Specifically, the track management unit 180 stores the track for each object, such as surrounding moving objects, and updates the track based on the processing results of the tracking filter unit 130. Some or all of the object data from the track management unit 180 is output to the track output unit 190.

[0039] The track output unit 190 outputs track information to the destination device 300 according to the requirements of the destination.

[0040] The output destination device 300 processes the track information output by the target tracking device 100 (100A). The output destination device 300 is, for example, a display device or a control device. If the output destination device 300 is a display device, it outputs to, for example, a display inside a nearby vehicle or a nearby display, and displays the information to the driver, pedestrians, etc. Alternatively, if the output destination device 300 is a control device for a moving object, some or all of the object data of the object included in the track information is output to a vehicle control unit consisting of a brake control device or a steering control device, etc., and this object data is used to perform control to maintain the distance between the vehicle and other moving objects (other vehicles), control to maintain the vehicle's lane, automatic driving control of the vehicle, etc.

[0041] Next, we will explain an example of processing in a target tracking device. First, we will explain an example of the processing performed by the target tracking device shown in Figure 1. Figure 4 is a flowchart showing an example of the process when the target tracking device is as shown in Figure 1. When the target tracking device 100 meets a pre-set target tracking start condition, such as when the target tracking device 100 starts to operate, it starts the process shown in Figure 4 (step ST1000 "start").

[0042] The target tracking device 100 then performs observation data processing (step ST1010 "Observed Values"). In the processing of step ST1010, the observation data processing unit 110 of the target tracking device 100 acquires the direction cosine for each observed value of the surrounding moving object detected based on the detection data of each of the multiple sensors. Specifically, the observation data processing unit 110 of the target tracking device 100 in this embodiment acquires the velocity vector and direction cosine for each observed value of the surrounding moving object detected based on the detection data of each of the multiple sensors.

[0043] The target tracking device 100 then performs correlation processing (step ST1020 “correlation”). In the processing of step ST1020, the correlation processing unit 120 of the target tracking device 100 calculates the correlation between the observed values ​​based on the direction cosines. Specifically, the correlation processing unit 120 of the target tracking device 100 in this embodiment calculates the correlation between the observed values ​​based on the velocity vector and the direction cosines.

[0044] The target tracking device 100 then performs integration processing (step ST1030 "integration"). In the processing of step ST1030, the integration unit (first integration unit) 140 of the target tracking device 100 integrates the correlated observed values. Specifically, the integration unit (first integration unit) 140 integrates the observed values ​​that have been correlated by the correlation processing unit 120.

[0045] The target tracking device 100 then performs identity determination processing (step ST1040 "Identity Determination (Identical?)"). In the process of step ST1040, the identity determination unit 150 of the target tracking device 100 determines whether the difference between the direction cosine of the observed value and the direction cosine of the stored track is within the range of the residual covariance matrix. Specifically, the identity determination unit 150 of the target tracking device 100 in this embodiment determines whether the integrated observed value and the stored track belong to the same object (surrounding moving object) by determining whether the difference between the velocity vector of the observed value and the velocity vector of the stored track, and the difference between the direction cosine of the observed value and the direction cosine of the stored track are within the range of the residual covariance matrix.

[0046] If the integrated observed values ​​and the stored track are not of the same object (surrounding moving object) (step ST1040 "Identity Determination (Identical?)" "NO"), the target tracking device 100 then performs alternative track processing (step ST1050 "Alternate Track"). In the processing of step ST1050, the identity determination unit 150 of the target tracking device 100 outputs information to the track management unit 180 indicating that it is a different track from the configuration in which the integrated observed values ​​are stored.

[0047] If the target tracking device 100 determines that the integrated observed values ​​and the stored track belong to the same object (surrounding moving object) (step ST1040 "Identity Determination (Identical?)" "YES"), it then performs a track integration process (step ST1060 "Track Integration"). In the process of step ST1060, the track integration unit (second integration unit) 160 of the target tracking device 100 integrates the observed values ​​and the track for the same object (surrounding moving object). The track integration unit (second integration unit) 160 outputs the integrated observed values ​​and the track if the identity determination unit 150 determines that the integrated observed values ​​and the stored track belong to the same object (surrounding moving object).

[0048] After the target tracking device 100 performs the track integration process in step ST1060, or after performing the separate track processing in step ST1050, it then performs a termination determination process (step ST1070 "Termination?"). In the termination determination process in step ST1070, a control unit (not shown) of the target tracking device 100 determines whether to terminate the processing of the target tracking device 100. The control unit (not shown) determines whether to terminate the processing of the target tracking device 100 according to, for example, an external termination command or execution program. If the control unit (not shown) determines that the target tracking device 100 has not finished processing (step ST1070 "Finished?" "NO"), the process proceeds to step ST1010, and the process is repeated from step ST1010. If a control unit (not shown) determines that the target tracking device 100 has finished processing (step ST1070 "Finish?" "YES"), the target tracking device 100 finishes the processing shown in Figure 4 (step ST1080 "Finish").

[0049] Next, we will explain an example of the processing of the target tracking device shown in Figure 2. Figure 5 is a flowchart showing an example of the process when the target tracking device is as shown in Figure 2. When the target tracking device 100 meets a pre-set target tracking start condition, such as when the target tracking device 100 starts to operate, it starts the process shown in Figure 5 (step ST1100 "start"). The processes from step ST1110 to step ST1150 in Figure 5 correspond to the processes from step ST1010 to step ST1050, which have already been explained, so their explanation is omitted here.

[0050] If the integrated observed values ​​and the stored track belong to the same object (surrounding moving object) (step ST1040 "Identity Determination (Identical?)" "YES"), the target tracking device 100 then performs a position correction determination process (step ST1160 "Position Correction?"). In the position correction determination process of step ST1160, the identity determination unit 150 of the target tracking device 100 determines whether to perform a position correction for the position included in the observed values. The determination criteria can be set as appropriate, for example, it may be a provision that specifies in advance whether to perform a position correction. If the identity determination unit 150 performs a position correction, it outputs the correlated observed values ​​or the integrated observed values ​​to the position correction unit 170. If the identity determination unit 150 does not perform a position correction, it outputs the integrated observed values ​​to the track integration unit (second integration unit) 160.

[0051] If the target tracking device 100 determines in the position correction determination process of step ST1160 to perform position correction (step ST1160 "Position Correction?" "YES"), it then performs the position correction process (step ST1170 "Position Correction"). In the position correction process of step ST1170, the position correction unit 170 of the target tracking device 100 replaces the position included in the observed values ​​before integration with the observed value that is considered to have higher accuracy, or corrects it to bring it closer to the correct position. For example, if the first sensor 11 and the second sensor 12 are a camera and a millimeter-wave radar device, the position correction unit 170 replaces the camera's observation position with the millimeter-wave observation position and outputs it to the track integration unit (second integration unit) 160 of the tracking filter unit 130.

[0052] After the target tracking device 100 determines in the position correction determination process of step ST1160 that it will not perform position correction (step ST1160 "Position Correction?" "NO"), or after performing the position correction process of step ST1170, it then performs the track integration process (step ST1180 "Track Integration"). In the process of step ST1180, the track integration unit (second integration unit) 160 of the target tracking device 100 performs the track integration process in the same manner as the track integration process of step ST1060 described earlier.

[0053] The processes from step ST1190 to step ST1200 in Figure 5 correspond to the processes from step ST1070 to step ST1080, which have already been explained, so their explanation is omitted here.

[0054] Next, we will explain a detailed example of the processing of the target tracking device shown in Figure 3. Figure 6 is a flowchart showing an example of the processing according to Embodiment 1 when the target tracking device is as shown in Figure 3. The process shown in Figure 6 is a target tracking method using a target tracking device.

[0055] In the target tracking device according to Embodiment 1, sensor observation values ​​obtained by a sensor are input, and a tracking filter such as a Kalman filter outputs a state vector smoothed value and a predicted value. The state vector of the surrounding vehicle to be estimated is “X k "Far away." State vector X k An example of this is a four-dimensional vector representing position and velocity in two-dimensional road coordinates. State vector X k This can be expressed by the following equation (1). TIFF0007876729000001.tif37166

[0056] The target tracking device 100 (100A) shown in Figure 6 starts the process shown in Figure 6 (step ST1210 "start") (operation start step) when a preset target tracking start condition is met, such as when the target tracking device 100 (100A) starts to operate.

[0057] Following the operation start step ST1210, the target tracking device 100 (100A) performs observation value input processing (step ST1220 "Observation Value Input") (Observation Value Input Step). In the observation value input processing of step ST1220, the observation value processing unit 110 of the target tracking device 100 (100A) receives camera observation value 1 input from the first sensor 11 and camera observation value 2 input from the second sensor 12.

[0058] The target tracking device 100 (100A) then performs a process to extract the velocity vector and direction cosine (step ST1230 “Velocity Vector, Direction Cosine Extraction”) (velocity vector, direction cosine extraction step). In the process of step ST1230, the observation value processing unit 110 of the target tracking device 100 (100A) extracts the velocity vector and direction cosine from the observed values.

[0059] The direction cosines u and v are calculated using the following equations (2) and (3). TIFF0007876729000002.tif24166

[0060] The observation vector based on the sensor information of the object (surrounding moving object, surrounding vehicle) obtained by the observation processing unit 110 is "Z k If we set this, the observation vector Z k This can be expressed by the following equation (4). TIFF0007876729000003.tif10166

[0061] The target tracking device 100 (100A) performs a correlation process (step ST1240 "correlation") following the velocity vector and direction cosine extraction process in step ST1230 (correlation step). In the correlation process of step ST1240, the correlation processing unit 120 of the target tracking device 100 (100A) performs correlation based on the velocity vector and direction cosine.

[0062] The target tracking device 100 (100A) executes a correlation determination process (step ST1250 "Correlation determination?") (correlation determination step) following the correlation process in step ST1240. In the correlation determination step of step ST1250, the correlation processing unit 120 of the tracking filter unit 130 in the target tracking device 100 (100A) determines whether the correlation of the observed values has been obtained.

[0063] When the correlation of the observed values is obtained (step ST1250 "Correlation determination?" "YES"), the target tracking device 100 (100A) then executes a process of integrating with an extended Kalman filter (extended KF) (step ST1270 "Integrate with extended KF") (integration step). In the integration step of step ST1270, the integration unit (first integration unit) 140 of the target tracking device 100 (100A) integrates the observed values with correlation by means of the extended Kalman filter process.

[0064] When the correlation of the observed values is not obtained (step ST1250 "Correlation determination?" "NO"), the target tracking device 100 (100A) then executes a separate track management process (step ST1260 "Separate track management") (separate track management step). In the separate track management step of step ST1260, the correlation processing unit 120 of the target tracking device 100 (100A) outputs the observed values without correlation to the track management unit 180 for management as a separate track. After executing the process of the separate track management step in step ST1260, the target tracking device 100 (100A) ends the process shown in FIG. 6. Alternatively, it may be repeated from the observed value input step in step ST1220.

[0065] At this time, using the state vector defined by Equation (1), the motion model of the target is described by the linear equation shown in Equation (5). X k+1 =Φ k X k +w k ···(5) “Φ kThe symbol represents the state transition matrix, and assuming uniform linear motion, the state transition matrix is ​​given by equation (6). TIFF0007876729000004.tif12166 Here, "I" represents the 2x2 identity matrix, and "0" represents the 2x2 zero matrix. Also, "Δt" represents the sampling interval.

[0066] Also, the second term on the right-hand side of equation (5) is “w k “ is the driving noise vector, which is a two-dimensional normally distributed white noise with mean 0. Assume that the expected value E, which represents the mean of this random variable, is given by equations (7) and (8). E[w k ]=0 ···(7) E[w k w k T ]=Q k ...(8) Equation (9) represents the observation model for the observed values ​​from the tracked target defined in Equation (4). "H" is the nonlinear observation matrix h(X k The Jacobian matrix obtained by linearizing ) is an example of equations (1) and (4), and is represented by equation (10). Z k =h(X k )+v k ...(9) TIFF0007876729000005.tif32166 The observation matrix H is the state vector X k This is the Jacobian matrix of h(x) for . TIFF0007876729000006.tif13166 “v k " is the sampling time t k Observation vector Z k This is the observed noise vector corresponding to the above, and is a 2D normally distributed white noise with a mean of 0, expressed by equations (12) and (13). Note that “R k " is the sampling time t kThe observed noise covariance matrix is ​​assumed to be independent of the motion model. Furthermore, the driving noise vector and the observed noise vector are assumed to be independent of each other. TIFF0007876729000007.tif26166 Based on the above definition, the tracking filter unit 130 operates according to the following equation to output smoothed and predicted values ​​of the state vector for each object (for each surrounding moving object, for each surrounding vehicle). TIFF0007876729000008.tif117166

[0067] The target tracking device 100 (100A) performs identity determination processing following the integration process in step ST1270 (step ST1280 "Identity Determination?") (identity determination step). In the identity determination step of step ST1280, the identity determination unit 150 of the target tracking device 100 (100A) determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object). Specifically, the identity determination unit 150 determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object) by, for example, determining whether the difference between the velocity vector of the camera observed values ​​and the velocity vector of the track is within the range of the residual covariance matrix.

[0068] In the identity determination step of step ST1280, if the target tracking device 100 (100A) determines that the integrated observed values ​​and the stored track belong to the same object (surrounding moving object) (step ST1280 "Identity determination?" "YES"), it then performs a position correction determination process (step ST1290 "Position correction?") (position correction determination step). In the position correction determination process of step ST1290, the identity determination unit 150 of the target tracking device 100 (100A) determines whether to perform a position correction.

[0069] If the target tracking device 100 (100A) determines to perform position correction (step ST1290 "Position Correction?" "YES"), it then performs position correction processing (step ST1300 "Position Correction") (position correction step). In the position correction processing of step ST1300, the position correction unit 170 of the target tracking device 100 (100A) replaces, for example, the position of the camera observation value with the position of the millimeter-wave radar observation value and outputs it to the track integration unit (second integration unit) 160.

[0070] If the target tracking device 100 (100A) determines that it does not need to perform position correction (step ST1290 "Position Correction?" "NO"), it then performs a smoothing value acquisition process (step ST1310 "Smoothing Value Acquisition") (Smoothing Value Acquisition Step). In the smoothing value acquisition process of step ST1310, the identity determination unit 150 of the target tracking device 100 (100A) acquires the smoothing value as an integrated observation value from the integration unit (first integration unit) 140 and outputs it to the track integration unit (second integration unit) 160.

[0071] If, in the identity determination step of step ST1280, the target tracking device 100 (100A) determines that the integrated observed values ​​and the stored track do not belong to the same object (surrounding moving object) (step ST1280 "Identity Determination?" "NO"), it then performs separate track management processing (step ST1260 "Separate Track Management"). In the processing of step ST1260, the identity determination unit 150 of the target tracking device 100 (100A) outputs information to the track management unit 180 indicating that the track is different from the configuration in which the integrated observed values ​​are stored. The track management unit 180 manages the integrated observation values ​​as a separate track from the stored track.

[0072] Here, the conditions for executing the integration process in step ST1270 (correlation determination in step ST1250) and the conditions for identity determination in step ST1280 are determined, for example, by whether the difference between the velocity vector of the camera observation and the velocity vector of the track falls within the range of the residual covariance matrix. This process follows equations (19) and (20). TIFF0007876729000009.tif10166 If equation (19) is true, we decide to integrate the observed values. TIFF0007876729000010.tif11166 If equation (20) is true, the observed values ​​are determined to be from different tracks.

[0073] Here, the “S” used in equations (19) and (20) k For this, you can use the residual covariance matrix calculated by equation (16), or you can apply the assumed error range parameters.

[0074] After the target tracking device 100 (100A) performs the position correction process in step ST1300, or after the smoothing value acquisition process in step ST1310, it then performs the track integration process (step ST1320 "track integration") (track integration step). In the track integration process of step ST1320, the track integration unit (second integration unit) 160 of the target tracking device 100 (100A) integrates the observed values ​​and the track for the same object (surrounding moving object). If the identity determination unit 150 determines that the integrated observed values ​​and the stored track belong to the same object (surrounding moving object), the track integration unit (second integration unit) 160 integrates the observed values ​​and the track and outputs them to the track management unit 180.

[0075] In the integration process in the track integration unit (second integration unit) 160, one of the following two methods is performed: The observed values, which have been correlated by the integration unit (first integration unit) 140, are integrated using an extended Kalman filter with direction cosines. or The position correction unit 170 replaces the camera observation position of the same target (same surrounding moving object, same surrounding vehicle) with the millimeter-wave observation position and integrates them using a linear Kalman filter.

[0076] The target tracking device 100 (100A) performs track output processing (step ST1330 "track output") following the track integration processing in step ST1320 (track output step). In the track output processing of step ST1330, the track output unit 190 of the target tracking device 100 (100A) acquires track information from the track management unit 180 and outputs the track to the output destination device 300 according to the requirements of the output destination.

[0077] After the target tracking device 100 (100A) executes the separate track management process in step ST1260, or after the track output process in step ST1330, it then executes a termination determination process (step ST1340 "Termination?"). In the termination determination process in step ST1340, a control unit (not shown) of the target tracking device 100 (100A) determines whether to terminate the processing of the target tracking device 100. The control unit (not shown) determines whether to terminate the processing of the target tracking device 100 (100A) according to, for example, an external termination command or execution program. If the control unit (not shown) determines that the target tracking device 100 (100A) has not finished processing (step ST1340 "Finished?" "NO"), the process proceeds to step ST1220, and the process is repeated from step ST1220. If a control unit (not shown) determines that the target tracking device 100 (100A) has finished processing (step ST1340 "Finished?" "YES"), the target tracking device 100 finishes the processing shown in Figure 6 (step ST1350 "Finished").

[0078] As described above, the target tracking device according to this embodiment determines identity by correlating with velocity vectors, thus preventing the integration of different targets located in the same direction. Furthermore, the target tracking device according to this embodiment does not use camera position information as a correlation parameter, but instead uses direction cosines to perform correlation and integration, thereby enabling accurate correlation and integration of surrounding vehicle information from multiple sensors. Furthermore, the target tracking device according to this embodiment does not use camera position information as correlation parameters, but instead uses only velocity vectors and direction cosines to perform correlation and integration, thereby enabling accurate correlation and integration of surrounding vehicle information from multiple sensors.

[0079] This embodiment shows an example configuration that includes the following: [1] An observation processing unit that acquires the direction cosine for each observed value of a surrounding moving object detected based on the detection data of multiple sensors, A correlation processing unit that calculates the correlation between observed values ​​based on the direction cosine, A first integration unit integrates the observed values ​​that have been correlated by the correlation processing unit, An identity determination unit that determines whether the difference between the direction cosine of the observed value and the direction cosine of the stored track is within the range of the residual covariance matrix, A second integration unit that integrates the observed value and the track and outputs the result when the identity determination unit determines that the result is within the range of the residual covariance matrix, A target tracking device equipped with [a specific feature]. This disclosure has the effect of providing a target tracking device that enables high-precision target tracking using sensor fusion.

[0080] This embodiment shows an example configuration that includes the following: [2] An observation processing unit that acquires the velocity vector and direction cosine for each observed value of a surrounding moving object detected based on the detection data of multiple sensors, A correlation processing unit that calculates the correlation between observed values ​​based on the velocity vector and the direction cosine, A first integration unit integrates the observed values ​​that have been correlated by the correlation processing unit, An identity determination unit that determines whether the difference between the velocity vector of the observed value and the velocity vector of the stored track, and the difference between the direction cosine of the observed value and the direction cosine of the stored track are within the range of the residual covariance matrix, A second integration unit that integrates the observed value and the track and outputs the result when the identity determination unit determines that the result is within the range of the residual covariance matrix, A target tracking device equipped with [a specific feature]. This disclosure provides a target tracking device that enables high-precision target tracking using sensor fusion by correlating observed values ​​using velocity vectors in addition to direction cosines.

[0081] This embodiment shows an example configuration that includes the following: [8] A target tracking method using a target tracking device, The observation value processing unit of the target tracking device acquires the direction cosine for each observed value of the surrounding moving object detected based on the detection data of each of the multiple sensors, The correlation processing unit of the target tracking device calculates the correlation between the observed values ​​based on the direction cosine, The first integration unit of the target tracking device integrates the observed values ​​that have been correlated by the correlation processing unit, The identity determination unit of the target tracking device determines whether the difference between the direction cosine of the observed value and the direction cosine of the stored track is within the range of the residual covariance matrix. The second integration unit of the target tracking device integrates the observed value and the track and outputs the result when the identity determination unit determines that the value is within the range of the residual covariance matrix. A target tracking method characterized by the following: This disclosure has the effect of providing a target tracking method that enables high-precision target tracking using sensor fusion.

[0082] This embodiment shows an example configuration that includes the following: [9] A target tracking method using a target tracking device, The observation processing unit of the target tracking device acquires the velocity vector and direction cosine for each observed value of the surrounding moving object detected based on the detection data of each of the multiple sensors. The correlation processing unit of the target tracking device performs correlation calculations between observed values ​​based on the velocity vector and the direction cosine. The first integration unit of the target tracking device integrates the observed values ​​that have been correlated by the correlation processing unit, The identity determination unit of the target tracking device determines whether the difference between the observed velocity vector and the stored track velocity vector, and the difference between the observed direction cosine and the stored track direction cosine, are within the range of the residual covariance matrix. The second integration unit of the target tracking device integrates the observed value and the track and outputs the result when the identity determination unit determines that the value is within the range of the residual covariance matrix. A target tracking method characterized by the following: This disclosure provides a target tracking method that enables high-precision target tracking using sensor fusion by correlating observed values ​​using velocity vectors in addition to direction cosines.

[0083] This embodiment further illustrates an example configuration including the following: [4] The second integration unit described above is Using the direction cosines of the observed values ​​correlated by the correlation processing unit, the observed values ​​and the track are integrated by an extended Kalman filter. A target tracking device according to any one of [1], [2], or [3], characterized in that [1], [2], or [3]. This disclosure further provides a target tracking device that enables high-precision target tracking using sensor fusion by configuring the device to integrate track data using direction cosines. Furthermore, by applying the above configuration to the target tracking system including the target tracking device, or to the target tracking method, the same effects as described above can be achieved.

[0084] This embodiment further illustrates an example configuration including the following: [5] The target tracking device according to claims 1, 2, and 3, characterized by integrating the track using the integrated smoothed values. The first integration unit described above is By calculating the smoothed value of the correlated observed values, the observed values ​​are integrated. The second integration unit described above is The observed values ​​and the track are integrated using the smoothed values. A target tracking device according to any one of [1], [2], or [3], characterized in that [1], [2], or [3]. This disclosure further provides a target tracking device that enables high-precision target tracking using sensor fusion by configuring the device to integrate track data using smoothed values. Furthermore, by applying the above configuration to the target tracking system including the target tracking device, or to the target tracking method, the same effects as described above can be achieved.

[0085] This embodiment further illustrates an example configuration including the following: [6] Replace the camera observation locations on the same vehicle with millimeter-wave observation locations and integrate them. The target tracking device according to claim 1, 2, or 3, characterized in that it is a target tracking device according to claim 1, 2, or 3. The second integration unit described above is The camera observation position, which is an observed value based on detection data from the imaging device, is replaced with the millimeter-wave observation position, which is an observed value based on detection data from the millimeter-wave radar device. The observed value and the track are then integrated using this replacement observed value. A target tracking device according to any one of [1], [2], or [3], characterized in that [1], [2], or [3]. This disclosure further provides a target tracking device that enables high-precision target tracking using sensor fusion by correcting the observed values ​​integrated with the flight path to more suitable values. Furthermore, by applying the above configuration to the target tracking system including the target tracking device, or to the target tracking method, the same effects as described above can be achieved.

[0086] This embodiment further illustrates an example configuration including the following: [7] The target tracking device according to claims 1, 2, and 3, characterized by integrating the positions of correlated observed values ​​using a linear Kalman filter. The second integration unit described above is Using the positions of surrounding moving objects included in the correlated observed values, the observed values ​​and the track are integrated by a linear Kalman filter. A target tracking device according to any one of [1], [2], [3], or [6], characterized by the following: This disclosure further provides a target tracking device that enables high-precision target tracking using sensor fusion by integrating the track using the position of each observed value. Furthermore, by applying the above configuration to the target tracking system including the target tracking device, or to the target tracking method, the same effects as described above can be achieved.

[0087] Embodiment 2. In the above-described embodiment 1, a method of correlation integration using the direction cosine and velocity vector for each observed value was explained. In Embodiment 2, when velocity vectors cannot be used in Embodiment 1, a method is described in which correlation is taken using direction cosines to integrate the vectors and then a determination of identity is made to decide whether to integrate them. In Embodiment 2, for components of Embodiment 2 that are the same as those of Embodiment 1 already described, the same component names are used with the same or similar reference numerals, and redundant explanations are omitted as appropriate.

[0088] Next, an example configuration of a target tracking device and a target tracking system including the target tracking device according to Embodiment 2 of this disclosure will be described with reference to Figures 1, 2, and 3. The target tracking system 1 is configured to include the target tracking device 100 according to Embodiment 2. Figure 3 shows the target tracking system 1 including the configuration of the target tracking device 100 shown in Figure 2. Embodiment 2 will be described assuming, as in Embodiment 1, that the sensor is composed only of a camera, or that the sensor is composed of a sensor with low positional accuracy. In the following description, the target tracking system 1 and target tracking device 100 according to Embodiment 2 will be referred to as target tracking system 1(1B) and target tracking device 100(100B) to distinguish them from the target tracking system 1 and target tracking device 100 according to other embodiments. The target tracking system 1(1B) shown in Figure 3 comprises a first sensor 11, a second sensor 12, a target tracking device 100(100B), and an output destination device 300.

[0089] The target tracking device 100 (100B) integrates observational values ​​based on detection data from multiple sensors and uses them to track the target. As shown in Figure 3, the target tracking device 100 (100B) is composed of an observation value processing unit 110, a correlation processing unit 120, a tracking filter unit 130, an integration unit (first integration unit) 140, an identity determination unit 150, a track integration unit (second integration unit) 160, a position correction unit 170, a track management unit 180, and a track output unit 190.

[0090] The observation processing unit 110 acquires detection data from multiple sensors and processes the observed value for each detection data. Based on the detection data from each of the multiple sensors, the observation processing unit 110 acquires the direction cosine for each observed value of the surrounding moving object detected.

[0091] The correlation processing unit 120 calculates the correlation between the observed values. The correlation processing unit 120 calculates the correlation between the observed values ​​based on the direction cosines.

[0092] The tracking filter unit 130 integrates the observed values ​​when correlated observed values ​​are input, and integrates the observed values ​​and the stored track if the integrated observed values ​​and the stored track are of the same object. The tracking filter unit 130 shown in Figure 3 is composed of an integration unit 140 (first integration unit), an identity determination unit 150, and a track integration unit 160 (second integration unit).

[0093] The integration unit 140 (first integration unit) of the tracking filter unit 130 integrates the observed values ​​that have been correlated by the correlation processing unit 120.

[0094] The identity determination unit 150 of the tracking filter unit 130 determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object). The identity determination unit 150 determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object) by, for example, determining whether the difference between the direction cosine of the observed value and the direction cosine of the stored track is within the range of the residual covariance matrix.

[0095] The track integration unit (second integration unit) 160 of the tracking filter unit 130 integrates the observed values ​​and track for the same object (surrounding moving object). The track integration unit 160 (second integration unit) integrates the observed values ​​and the track and outputs the result when the identity determination unit determines that they are within the range of the residual covariance matrix. The track integration unit 160 (second integration unit) integrates the observed values ​​and the track using an extended Kalman filter process, with the direction cosines of the observed values ​​correlated by the correlation processing unit. Alternatively, the track integration unit 160 (second integration unit) integrates the observed values ​​and the track using the smoothed values. Alternatively, the track integration unit 160 (second integration unit) integrates the observed values ​​and the track using a linear Kalman filter process, using the positions of surrounding moving objects included in the correlated observed values.

[0096] The position correction unit 170 corrects the position indicated by the correlated observed values. The position correction unit 170 corrects the position included in the observed values ​​before integration with the track. Specifically, the position correction unit 170 corrects the position included in the observed values ​​before integration with the track when the correlated observed values ​​are the same moving object as the moving object in the track stored in the track management unit 180. For example, the position correction unit 170 replaces the camera observation position with the millimeter-wave observation position and outputs it to the track integration unit (second integration unit) 160 of the tracking filter unit 130. Specifically, the position correction unit 170 replaces the camera observation position, which is an observed value based on the detection data of the imaging device, with the millimeter-wave observation position, which is an observed value based on the detection data of the millimeter-wave radar device.

[0097] The track integration unit (second integration unit) 160 can integrate the track with the observed values ​​corrected by the position correction unit 170. The track integration unit (second integration unit) 160 integrates the track with the observed values ​​using observed values ​​in which the position (camera observation position), which is an observed value based on the detection data of the imaging device, has been replaced with the position (millimeter-wave observation position), which is an observed value based on the detection data of the millimeter-wave radar device. The track integration unit (second integration unit) 160 integrates the track with the observed values ​​by linear Kalman filtering using the positions of surrounding moving objects included in the correlated observed values. The track integration unit (second integration unit) 160 may be configured to have the above-described functions in addition to the functions of the track integration unit (second integration unit) 160 already described. In this case, the track integration unit (second integration unit) 160 may be configured to, for example, determine whether to perform position correction.

[0098] The track management unit 180 can be configured in the same way as the track management unit 180 already described, so its explanation is omitted here. The track output unit 190 can be configured in the same way as the track output unit 190 already described, so its description is omitted here.

[0099] In Embodiment 2, the target tracking device 100 (100B) takes sensor observation values ​​obtained by the sensor as input and outputs a state vector smoothed value and a predicted value using a tracking filter such as a Kalman filter. The state vector of the surrounding moving object (surrounding vehicle) to be estimated is “X k "Far away." State vector X k An example of this is a two-dimensional vector representing a position in two-dimensional road coordinates.

[0100] State vector X k This can be expressed by the following equation (21). X k =[x k y k ] T ···(twenty one) Here, “x k " represents the vertical position of the surrounding moving object (surrounding vehicle), and "y k The symbol " represents the lateral position of the surrounding moving object (surrounding vehicle), and the entirety is represented by vector X. k This will be indicated as follows.

[0101] Furthermore, the observation vector of the sensor information of the surrounding moving object (surrounding vehicle) obtained in the observation value processing unit 110 is set to “Z k If we set it as ", the observation vector Z k It is expressed by the following equation (22). Z k =[uv] T ···(twenty two)

[0102] The operation of the target tracking device 100 (100B) according to this second embodiment will be described below with reference to Figure 7. A detailed example of the processing of the target tracking device according to Embodiment 2 of this disclosure will be described. Figure 7 is a flowchart showing an example of the processing according to Embodiment 2 in the case of the target tracking device shown in Figure 3.

[0103] The target tracking device 100 (100B) starts the process shown in Figure 7 when it meets a pre-set target tracking start condition, such as when the target tracking device 100 (100B) starts to operate (step ST2210 "start") (operation start step).

[0104] Following the operation start step ST2210, the target tracking device 100 (100B) performs observation value input processing (step ST2220 "Observation Value Input") (Observation Value Input Step). In the observation value input processing of step ST2220, the observation value processing unit 110 of the target tracking device 100 (100B) receives camera observation value 1 input from the first sensor 11 and camera observation value 2 input from the second sensor 12.

[0105] The target tracking device 100 (100B) performs direction cosine extraction processing (step ST2230 "direction cosine extraction") following the observation value input processing in step ST2220 (direction cosine extraction step). In the direction cosine extraction processing of step ST2230, the observation value processing unit 110 of the target tracking device 100 (100B) extracts the direction cosine from the observation value and outputs it to the correlation processing unit 120.

[0106] The target tracking device 100 (100B) performs a correlation process (step ST2240 "correlation") following the direction cosine extraction process in step ST2230 (correlation step). In the correlation process of step ST2240, the correlation processing unit 120 of the target tracking device 100 (100B) performs correlation based on the direction cosines.

[0107] The target tracking device 100 (100B) then performs a correlation determination process (step ST2250 "Correlation Determination?") (correlation determination step). In the correlation determination process of step ST2250, the correlation processing unit 120 of the target tracking device 100 (100B) determines whether the observed values ​​have been correlated.

[0108] If the target tracking device 100 (100B) fails to find a correlation between the observed values ​​(step ST2250 "Correlation Determination?" "NO"), it then performs a separate track management process (step ST2260 "Separate Track Management") (Separate Track Management Step). In the separate track management process of step ST2260, the correlation processing unit 120 of the target tracking device 100 (100B) outputs to the track management unit 180 to manage the observed values ​​for which no correlation was found as separate tracks. After executing the processing of the separate track management step in step ST2260, the target tracking device 100 (100B) proceeds to the termination determination process in step ST2340 shown in Figure 7. Alternatively, the process may be repeated starting from the observed value input step in step ST2220.

[0109] If the target tracking device 100 (100B) finds a correlation between the observed values ​​(step ST2250 "Correlation Determination?" "YES"), it then performs a process to integrate the values ​​using an extended Kalman filter (extended KF) (step ST2270 "Integrate with Extended KF") (integration step). In the integration process of step ST2270, the integration unit (first integration unit) 140 of the target tracking device 100 (100B) integrates the correlated observed values ​​using the extended Kalman filter.

[0110] The target tracking device 100 (100B) performs identity determination processing following the integration processing in step ST2270 (step ST2280 "Identity Determination?") (identity determination step). In the identity determination processing of step ST2280, the identity determination unit 150 of the target tracking device 100 (100B) determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object). The identity determination unit 150 determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object) by determining whether the difference between the direction cosine of the observed values ​​and the direction cosine of the stored track is within the range of the residual covariance matrix. Alternatively, the identity determination unit 150 determines that the objects are the same target (surrounding moving object, surrounding vehicle) if, for example, the difference between the integrated observation value and the latest observation value in the stored track is within a preset threshold. The identity determination unit 150 also determines that the objects are the same target (surrounding moving object, surrounding vehicle) if, for example, the difference between the integrated observation value and the latest observation value in the stored track is outside a preset threshold.

[0111] If the target tracking device 100 (100B) determines in the identity determination process of step ST2280 that the integrated observed values ​​and the stored track do not belong to the same object (surrounding moving object) (step ST2280 "Identity Determination?" "NO"), it then executes a separate track management process (step ST2260 "Separate Track Management"). In the separate track management process of step ST2260, the identity determination unit 150 of the target tracking device 100 (100B) outputs information to the track management unit 180 indicating that the track is separate from the configuration in which the integrated observed values ​​are stored. The track management unit 180 manages the track as separate from the track in which the integrated observed values ​​are stored.

[0112] In the identity determination process of step ST2280, if the target tracking device 100 (100B) determines that the integrated observed values ​​and the stored track belong to the same object (surrounding moving object) (step ST2280 "Identity determination?" "YES"), it then performs a position correction determination process (step ST2290 "Position correction?") (position correction determination step). In the position correction determination process of step ST2290, the identity determination unit 150 of the target tracking device 100 (100B) determines whether to perform a position correction.

[0113] If the target tracking device 100 (100B) determines to perform position correction (step ST2290 "Position Correction?" "YES"), it then performs position correction processing (step ST2300 "Position Correction") (position correction step). In the position correction processing of step ST2300, the position correction unit 170 of the target tracking device 100 (100B) replaces, for example, the position of the camera observation value with the position of the millimeter-wave radar observation value and outputs it to the track integration unit (second integration unit) 160.

[0114] If the target tracking device 100 (100B) determines that it does not need to perform position correction (step ST2290 "Position Correction?" "NO"), it then performs a smoothing value acquisition process (step ST2310 "Smoothing Value Acquisition") (Smoothing Value Acquisition Step). In the smoothing value acquisition process of step ST2310, the identity determination unit 150 of the target tracking device 100 (100B) acquires the smoothing value as an integrated observation value from the integration unit (first integration unit) 140 and outputs it to the track integration unit (second integration unit) 160.

[0115] After the target tracking device 100 (100B) performs the position correction process in step ST2300, or after the smoothing value acquisition process in step ST2310, it then performs the track integration process (step ST2320 "track integration") (track integration step). In the track integration process of step ST2320, the track integration unit (second integration unit) 160 of the target tracking device 100 (100B) integrates the observed values ​​and the track for the same object (surrounding moving object). If the identity determination unit 150 determines that the integrated observed values ​​and the stored track belong to the same object (surrounding moving object), the track integration unit (second integration unit) 160 integrates the observed values ​​and the track and outputs them to the track management unit 180.

[0116] In the integration process in the track integration unit (second integration unit) 160, one of the following two methods is performed: The observed values, which have been correlated by the integration unit (first integration unit) 140, are integrated using an extended Kalman filter with direction cosines. or The position correction unit 170 replaces the camera observation position of the same target (same surrounding moving object, same surrounding vehicle) with the millimeter-wave observation position and integrates them using a linear Kalman filter.

[0117] The target tracking device 100 (100B) performs track output processing (step ST2330 "track output") following the track integration processing in step ST2320 (track output step). In the track output processing of step ST2330, the track output unit 190 of the target tracking device 100 (100B) acquires track information from the track management unit 180 and outputs the track to the output destination device 300 according to the requirements of the output destination.

[0118] After the target tracking device 100 (100B) executes the separate track management process in step ST2260, or after executing the track output process in step ST2330, it then executes a termination determination process (step ST2340 "Termination?"). In the termination determination process in step ST2340, a control unit (not shown) of the target tracking device 100 (100B) determines whether to terminate the processing of the target tracking device 100. The control unit (not shown) determines whether to terminate the processing of the target tracking device 100 (100B) according to, for example, an external termination command or execution program. If the control unit (not shown) determines that the target tracking device 100 (100B) has not finished processing (step ST2340 "Finished?" "NO"), the process proceeds to step ST2220, and the process is repeated from step ST2220. If a control unit (not shown) determines that the target tracking device 100 (100B) has finished processing (step ST2340 "Finished?" "YES"), the target tracking device 100 finishes the processing shown in Figure 7 (step ST2350 "Finished").

[0119] As described above, the target tracking device according to this embodiment uses direction cosines to determine identity, thus preventing the integration of different targets located in the same direction. Furthermore, the target tracking device according to this embodiment does not use camera position information as a correlation parameter, but instead uses direction cosines to perform correlation and integration, thereby enabling accurate correlation and integration of surrounding vehicle information from multiple sensors.

[0120] This embodiment shows an example configuration that includes the following: [1] An observation processing unit that acquires the direction cosine for each observed value of a surrounding moving object detected based on the detection data of multiple sensors, A correlation processing unit that calculates the correlation between observed values ​​based on the direction cosine, A first integration unit integrates the observed values ​​that have been correlated by the correlation processing unit, An identity determination unit that determines whether the difference between the direction cosine of the observed value and the direction cosine of the stored track is within the range of the residual covariance matrix, A second integration unit that integrates the observed value and the track and outputs the result when the identity determination unit determines that the result is within the range of the residual covariance matrix, A target tracking device equipped with [a specific feature]. This disclosure has the effect of providing a target tracking device that enables high-precision target tracking using sensor fusion.

[0121] This embodiment shows an example configuration that includes the following: [8] A target tracking method using a target tracking device, The observation value processing unit of the target tracking device acquires the direction cosine for each observed value of the surrounding moving object detected based on the detection data of each of the multiple sensors, The correlation processing unit of the target tracking device calculates the correlation between the observed values ​​based on the direction cosine, The first integration unit of the target tracking device integrates the correlated observed values, The identity determination unit of the target tracking device determines whether the difference between the direction cosine of the observed value and the direction cosine of the stored track is within the range of the residual covariance matrix. The second integration unit of the target tracking device integrates the observed value and the track and outputs the result when the identity determination unit determines that the value is within the range of the residual covariance matrix. A target tracking method characterized by the following: This disclosure has the effect of providing a target tracking method that enables high-precision target tracking using sensor fusion.

[0122] This embodiment further illustrates an example configuration including the following: [4] The second integration unit described above is Using the direction cosines of the observed values ​​correlated by the correlation processing unit, the observed values ​​and the track are integrated by an extended Kalman filter. A target tracking device according to any one of [1], [2], or [3], characterized in that [1], [2], or [3]. This disclosure further provides a target tracking device that enables high-precision target tracking using sensor fusion by configuring the device to integrate track data using direction cosines. Furthermore, by applying the above configuration to the target tracking system including the target tracking device, or to the target tracking method, the same effects as described above can be achieved.

[0123] This embodiment further illustrates an example configuration including the following: [5] The first integration unit described above is By calculating the smoothed value of the correlated observed values, the observed values ​​are integrated. The second integration unit described above is The observed values ​​and the track are integrated using the smoothed values. A target tracking device according to any one of [1], [2], or [3], characterized in that [1], [2], or [3]. This disclosure further provides a target tracking device that enables high-precision target tracking using sensor fusion by configuring the device to integrate track data using smoothed values. Furthermore, by applying the above configuration to the target tracking system including the target tracking device, or to the target tracking method, the same effects as described above can be achieved.

[0124] This embodiment further illustrates an example configuration including the following: [6] The second integration unit described above is The camera observation position, which is an observed value based on detection data from the imaging device, is replaced with the millimeter-wave observation position, which is an observed value based on detection data from the millimeter-wave radar device. The observed value and the track are then integrated using this replacement observed value. A target tracking device according to any one of [1], [2], or [3], characterized in that [1], [2], or [3]. This disclosure further provides a target tracking device that enables high-precision target tracking using sensor fusion by correcting the observed values ​​integrated with the flight path to more suitable values. Furthermore, by applying the above configuration to the target tracking system including the target tracking device, or to the target tracking method, the same effects as described above can be achieved.

[0125] This embodiment further illustrates an example configuration including the following: [7] The second integration unit described above is Using the positions of surrounding moving objects included in the correlated observed values, the observed values ​​and the track are integrated by a linear Kalman filter. A target tracking device according to any one of [1], [2], [3], or [6], characterized by the following: This disclosure further provides a target tracking device that enables high-precision target tracking using sensor fusion by integrating the track using the position of each observed value. Furthermore, by applying the above configuration to the target tracking system including the target tracking device, or to the target tracking method, the same effects as described above can be achieved.

[0126] Embodiment 3. In Embodiment 1 described above, a method of correlation integration using velocity vectors and direction cosines was explained. In Embodiment 2 described above, a method of correlation integration using direction cosines was explained. Embodiment 3 describes a configuration example suitable for cases where the camera observation values ​​include the width and depth of the target, and where observation values ​​from different sensors with high positional accuracy, such as millimeter-wave radar, can be obtained. In Embodiment 3, for components related to Embodiment 3 that are the same as those related to Embodiment 1 or Embodiment 2 already described, the same component names are used with the same or similar reference numerals, and redundant explanations are omitted as appropriate.

[0127] Next, an example of the configuration of a target tracking device and a target tracking system including the target tracking device according to Embodiment 3 of this disclosure will be described with reference to Figure 3. The target tracking system 1 includes the target tracking device 100 according to Embodiment 3. Figure 3 shows the target tracking system 1 including the configuration of the target tracking device 100 shown in Figure 2. In the following description, the target tracking system 1 and target tracking device 100 according to Embodiment 3 will be referred to as target tracking system 1(1C) and target tracking device 100(100C) to distinguish them from the target tracking system 1 and target tracking device 100 according to other embodiments. The target tracking system 1(1C) shown in Figure 3 comprises a first sensor 11, a second sensor 12, a target tracking device 100(100C), and an output device 300.

[0128] The first sensor 11 and the second sensor 12 are, for example, an imaging device and a millimeter-wave radar device. In this embodiment, it is assumed that one sensor is a camera with low positional accuracy, and the other sensor is a sensor with high positional accuracy, such as a millimeter-wave radar.

[0129] The target tracking device 100 (100C) integrates observational values ​​based on detection data from multiple sensors and uses them to track the target. As shown in Figure 3, the target tracking device 100 (100C) is configured to include an observation value processing unit 110, a correlation processing unit 120, a tracking filter unit 130, a position correction unit 170, a track management unit 180, and a track output unit 190.

[0130] The observation processing unit 110 acquires detection data from multiple sensors and processes the observed value for each detection data. Based on the detection data from the imaging device, the observation processing unit 110 acquires the width and depth of each surrounding moving object detected.

[0131] The correlation processing unit 120 calculates the correlation between the observed values. The correlation processing unit 120 sets a coarse gate based on the width and depth of the surrounding moving object, and for each coarse gate, calculates the correlation between the observed values ​​based on the position of each observed value of the moving object detected based on the detection data of the millimeter-wave radar device.

[0132] The tracking filter unit 130 integrates the observed values ​​when correlated observed values ​​are input, and integrates the observed values ​​and the stored track if the integrated observed values ​​and the stored track are of the same object. The tracking filter unit 130 shown in Figure 3 is composed of an integration unit (first integration unit) 140, an identity determination unit 150, and a track integration unit (second integration unit) 160.

[0133] The integration unit (first integration unit) 140 of the tracking filter unit 130 integrates the observed values ​​that have been correlated by the correlation processing unit 120.

[0134] The identity determination unit 150 of the tracking filter unit 130 determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object). Specifically, the identity determination unit 150 determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object) by, for example, determining whether the difference between the position of the observed value and the position of the track is within the range of the residual covariance matrix.

[0135] The track integration unit (second integration unit) 160 of the tracking filter unit 130 integrates the observed values ​​and the track for the same object (surrounding moving object). Specifically, the track integration unit (second integration unit) 160 integrates the observed values ​​and the track and outputs the result when the identity determination unit 150 determines that the values ​​are within the range of the residual covariance matrix. The track integration unit 160 (second integration unit) integrates the observed values ​​and the track using an extended Kalman filter process, with the direction cosines of the observed values ​​correlated by the correlation processing unit 120. Alternatively, the track integration unit 160 (second integration unit) integrates the observed values ​​and the track using the smoothed values. Alternatively, the track integration unit 160 (second integration unit) integrates the observed values ​​and the track using a linear Kalman filter process, using the positions of surrounding moving objects included in the correlated observed values.

[0136] The position correction unit 170 corrects the position indicated by the correlated observed values. The position correction unit 170 corrects the position included in the observed values ​​before integration with the track. Specifically, the position correction unit 170 corrects the position included in the observed values ​​before integration with the track when the correlated observed values ​​are the same moving object as the moving object in the track stored in the track management unit 180. For example, the position correction unit 170 replaces the camera observation position with the millimeter-wave observation position and outputs it to the track integration unit (second integration unit) 160 of the tracking filter unit 130. Specifically, the position correction unit 170 replaces the camera observation position, which is an observed value based on the detection data of the imaging device, with the millimeter-wave observation position, which is an observed value based on the detection data of the millimeter-wave radar device.

[0137] The track integration unit (second integration unit) 160 can integrate the track with the observed values ​​corrected by the position correction unit 170. The track integration unit (second integration unit) 160 integrates the track with the observed values ​​using observed values ​​in which the position (camera observation position), which is an observed value based on the detection data of the imaging device, has been replaced with the position (millimeter-wave observation position), which is an observed value based on the detection data of the millimeter-wave radar device. The track integration unit (second integration unit) 160 integrates the track with the observed values ​​by linear Kalman filtering using the positions of surrounding moving objects included in the correlated observed values. The track integration unit (second integration unit) 160 may be configured to have the above-described functions in addition to the functions of the track integration unit (second integration unit) 160 already described. In this case, the track integration unit (second integration unit) 160 may be configured to, for example, determine whether to perform position correction.

[0138] The track management unit 180 can be configured in the same way as the track management unit 180 already described, so its explanation is omitted here. The track output unit 190 can be configured in the same way as the track output unit 190 already described, so its description is omitted here.

[0139] In Embodiment 3, the target tracking device 100 (100C) receives sensor observation values ​​obtained by the sensor and outputs a state vector smoothed value and a predicted value using a tracking filter such as a Kalman filter. The state vector of the surrounding vehicle to be estimated is “X k "Far away." State vector X k An example of this is a two-dimensional vector representing a position in two-dimensional road coordinates. State vector X k This can be expressed by the following equation (23). X k =[x k y k ] T ···(twenty three) Here, “x k " indicates the vertical position of surrounding vehicles, "y k The symbol " represents the lateral position of surrounding vehicles, and the entirety is represented by vector X. k This will be indicated as follows. Furthermore, the observation vector of the sensor information of the surrounding vehicle obtained by the observation value processing unit 110 of Embodiment 3 is “Z k If we set it as ", the observation vector Z k This can be expressed by the following equations (24) and (25). Z k =HX k +v k ···(twenty four) Z k =[x k y k ] T ···(twenty five) "H" is the observation matrix, and if it satisfies equations (24) and (25), then equation (26) is true. H=I ···(26)

[0140] Next, a detailed example of the processing of the target tracking device according to Embodiment 3 of this disclosure will be described. Figure 8 is a flowchart showing an example of the processing according to Embodiment 3 in the case of the target tracking device shown in Figure 3. When the target tracking device 100(100C) meets a pre-set target tracking start condition, such as when the target tracking device 100(100B) starts to operate, it starts the process shown in Figure 8 (step ST3210 "start") (operation start step).

[0141] Following the operation start step ST3210, the target tracking device 100 (100C) performs observation value input processing (step ST3220 "Observation Value Input") (Observation Value Input Step). In the observation value input processing of step ST3220, the observation value processing unit 110 of the target tracking device 100 (100C) receives millimeter-wave radar observation values ​​input from the first sensor 11 and camera observation values ​​input from the second sensor 12.

[0142] The target tracking device 100 (100C) performs camera coarse gate processing (step ST3230 "camera coarse gate") following the observation value input processing in step ST3220 (camera coarse gate step). In the camera coarse gate processing of step ST3230, the observation value processing unit 110 of the target tracking device 100 (100C) acquires the width and depth of each surrounding moving object detected based on the detection data of the imaging device. The observation value processing unit 110 extracts the width and depth from the camera observation values, forms a coarse gate, extracts the millimeter-wave radar observation values ​​within the gate, and outputs them to the correlation processing unit 120.

[0143] An example of a state vector is a two-dimensional position vector. This state vector is considered to be based on two-dimensional road coordinates. In addition, the observation vector of the sensor information of surrounding vehicles obtained by the observation processing unit 110 is "Z k If we set it as ", the observation vector Z k This can be expressed by the following equations (27) and (28). X k =[x k y k ] T ...(27) Z k =[x k y k ] T ...(28) Here, “x k " indicates the vertical position of surrounding vehicles, "y k The dots represent the lateral position of surrounding vehicles. Furthermore, these dot symbols represent their vertical and lateral speeds, and the whole is represented by vector X. k This will be indicated as follows.

[0144] Width x from camera observations left , x right , depth y near , y far This is used to create a coarse gate. The process for determining whether to extract radar observation values ​​within the gate follows equations (29) and (30). x left ≤ x k ≤ x right ...(29) y near ≤ y k ≤ y far ...(30) Radar observations that satisfy the conditions of equations (29) and (30) are determined to be within the coarse gate and are integrated into the next tracking filter as radar observations with positional correlation from the same target. If it is determined that the target is not within the rough gate area, it will be considered a different target and managed as a separate track.

[0145] The target tracking device 100 (100C) performs correlation processing (step ST3240 "correlation") following the camera coarse gate processing in step ST3230 (correlation step). In the correlation processing of step ST3240, the correlation processing unit 120 of the target tracking device 100 (100C) performs correlation at the millimeter-wave radar observation position. Specifically, the correlation processing unit 120 sets a coarse gate based on the width and depth of the surrounding moving object. For each coarse gate, the correlation processing unit 120 performs correlation between the observed values ​​based on the position of each observed value of the moving object detected based on the detection data of the millimeter-wave radar device.

[0146] The target tracking device 100 (100C) performs a correlation determination process following the correlation processing in step ST3240 (step ST3250 "Correlation Determination?") (correlation determination step). In the correlation determination process of step ST3250, the correlation processing unit 120 of the target tracking device 100 (100C) determines whether the observed values ​​have been correlated.

[0147] If the target tracking device 100 (100C) fails to find a correlation between the observed values ​​(step ST3250 "Correlation Determination?" "NO"), it then performs a separate track management process (step ST3260 "Separate Track Management") (Separate Track Management Step). In the separate track management process of step ST3260, the correlation processing unit 120 of the target tracking device 100 (100C) outputs to the track management unit 180 to manage the observed values ​​for which no correlation was found as separate tracks. The target tracking device 100 (100C) proceeds to the termination determination process in step ST3340 after executing the processing of the separate track management step in step ST3260. Alternatively, the process may be repeated starting from the observed value input step in step ST3220.

[0148] If the target tracking device 100 (100C) finds that the observed values ​​are correlated (step ST3250 "Correlation Determination?" "YES"), it then performs a process to integrate them using a linear Kalman filter (linear KF) (step ST3270 "Integrate with Linear KF") (integration step). In the integration process of step ST3270, the integration unit (first integration unit) 140 of the target tracking device 100 (100C) integrates the observed values ​​that have been correlated by the correlation processing unit 120. Specifically, the integration unit (first integration unit) 140 integrates the correlated observed values ​​by processing them with a linear Kalman filter, for example.

[0149] The target tracking device 100 (100C) performs identity determination processing following the integration processing in step ST3270 (step ST3280 "Identity Determination?") (identity determination step). In the identity determination processing of step ST3280, the identity determination unit 150 of the target tracking device 100 (100C) determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object). The identity determination unit 150 determines whether the integrated observed values ​​and the stored track belong to the same object (surrounding moving object) by determining whether the difference between the position of the observed value and the position of the track is within the range of the residual covariance matrix. Alternatively, the identity determination unit 150 determines that the objects are the same target (surrounding moving object, surrounding vehicle) if, for example, the difference between the integrated observation value and the latest observation value in the stored track is within a preset threshold. The identity determination unit 150 also determines that the objects are the same target (surrounding moving object, surrounding vehicle) if, for example, the difference between the integrated observation value and the latest observation value in the stored track is outside a preset threshold.

[0150] If the target tracking device 100 (100C) determines in the identity determination process of step ST3280 that the integrated observed values ​​and the stored track belong to the same object (surrounding moving object) (step ST3280 "Identity determination?" "YES"), it then performs a smoothed value acquisition process (step ST3300 "Smoothed value acquisition") (smoothed value acquisition step). In the smoothed value acquisition process of step ST3300, the identity determination unit 150 of the target tracking device 100 (100C) acquires the smoothed values ​​as integrated observed values ​​from the integration unit (first integration unit) 140 and outputs them to the track integration unit (second integration unit) 160.

[0151] If the target tracking device 100 (100C) determines in the identity determination process of step ST3280 that the integrated observed values ​​and the stored track do not belong to the same object (surrounding moving object) (step ST3280 "Identity Determination?" "NO"), it then executes a separate track management process (step ST3260 "Separate Track Management"). In the separate track management process of step ST2260, the identity determination unit 150 of the target tracking device 100 (100C) outputs information to the track management unit 180 indicating that the track is separate from the configuration in which the integrated observed values ​​are stored. The track management unit 180 manages the track as separate from the track in which the integrated observed values ​​are stored.

[0152] The target tracking device 100 (100C) performs a process (step ST3320 "track integration") (track integration step) following the smoothing value acquisition process in step ST3300. In the track integration process of step ST3320, the track integration unit (second integration unit) 160 of the target tracking device 100 (100C) integrates the observed values ​​and the track for the same object (surrounding moving object). If the identity determination unit 150 determines that the integrated observed values ​​and the stored track belong to the same object (surrounding moving object), the track integration unit (second integration unit) 160 integrates the observed values ​​and the track and outputs them to the track management unit 180.

[0153] The target tracking device 100 (100C) performs track output processing (step ST3330 "track output") following the track integration processing in step ST3320 (track output step). In the track output processing of step ST3330, the track output unit 190 of the target tracking device 100 (100C) acquires track information from the track management unit 180 and outputs the track to the output destination device 300 according to the requirements of the output destination.

[0154] After the target tracking device 100(100C) executes the separate track management process in step ST3260, or after the track output process in step ST3330, it then executes a termination determination process (step ST3340 "Termination?"). In the termination determination process in step ST3340, a control unit (not shown) of the target tracking device 100(100C) determines whether to terminate the processing of the target tracking device 100(100C). The control unit (not shown) determines whether to terminate the processing of the target tracking device 100(100C) according to, for example, an external termination command or execution program. If the control unit (not shown) determines that the target tracking device 100 (100C) has not finished processing (step ST3340 "Finished?" "NO"), the process proceeds to step ST3220, and the process is repeated from step ST3220. If the control unit (not shown) determines that the target tracking device 100 (100C) has finished processing (step ST3340 "Finish?" "YES"), the target tracking device 100 (100C) finishes the processing shown in Figure 8 (step ST3350 "Finish").

[0155] As described above, the target tracking device according to this embodiment determines identity by correlating radar observation values ​​within the camera's coarse gate, thus preventing the integration of different targets located in the same direction. In addition, the target tracking device according to the present embodiment uses, as correlation parameters, a rough gate obtained by using the vehicle size (depth and width of an object) obtained by a camera instead of the position information of the camera, applies it to the millimeter-wave observation position to determine whether it is inside or outside the gate, and correlates and integrates using the position of the millimeter-wave radar observation value. Therefore, the width and depth information of an object (surrounding moving object, surrounding vehicle) from the first sensor (specific example: camera) and the object information (surrounding moving object information, surrounding vehicle information) from the second sensor (specific example: millimeter-wave radar) can be correctly correlated and integrated.

[0156] The present embodiment shows a form example including the following configuration. [3] An observation value processing unit that acquires the width and depth of each surrounding moving object detected based on the detection data of the imaging device, A correlation processing unit that sets a rough gate based on the width and depth of the surrounding moving object, and correlates the observation values for each moving object detected based on the detection data of the millimeter-wave radar device for each rough gate based on the position of the observation value, A first integration unit that integrates the observation values correlated by the correlation processing unit, An identity determination unit that determines whether the difference between the position of the observation value and the position of the track is within the range of the residual covariance matrix, A second integration unit that integrates and outputs the observation value and the track when it is determined by the identity determination unit that it is within the range of the residual covariance matrix, A target tracking device comprising: As a result, the present disclosure can provide a target tracking device capable of realizing high-precision target tracking using sensor fusion, and has the effect of being able to achieve this.

[0157] The present embodiment shows a form example including the following configuration.

[10] A target tracking method by a target tracking device, The observation value processing unit of the target tracking device acquires the width and depth of each surrounding moving object detected based on the detection data of the imaging device, The correlation processing unit of the target tracking device sets a coarse gate based on the width and depth of the surrounding moving object, and for each coarse gate, it calculates the correlation between the observed values ​​based on the position of each observed value of the moving object detected based on the detection data of the millimeter-wave radar device. The first integration unit of the target tracking device integrates the observed values ​​that have been correlated by the correlation processing unit, The identity determination unit of the target tracking device determines whether the difference between the position of the observed value and the position of the track is within the range of the residual covariance matrix. The second integration unit of the target tracking device integrates the observed value and the track and outputs the result when the identity determination unit determines that the value is within the range of the residual covariance matrix. A target tracking method characterized by the following: This disclosure has the effect of providing a target tracking method that enables high-precision target tracking using sensor fusion.

[0158] This embodiment further illustrates an example configuration including the following: [4] The second integration unit described above is Using the direction cosines of the observed values ​​correlated by the correlation processing unit, the observed values ​​and the track are integrated by an extended Kalman filter. A target tracking device according to any one of claims 1 to 3. This disclosure further provides a target tracking device that enables high-precision target tracking using sensor fusion by configuring the device to integrate track data using direction cosines. Furthermore, by applying the above configuration to the target tracking system including the target tracking device, or to the target tracking method, the same effects as described above can be achieved.

[0159] This embodiment further illustrates an example configuration including the following: [5] The first integration unit described above is By calculating the smoothed value of the correlated observed values, the observed values ​​are integrated. The second integration unit described above is The observed values ​​and the track are integrated using the smoothed values. A target tracking device according to any one of [1], [2], or [3], characterized in that... This disclosure further provides a target tracking device that enables high-precision target tracking using sensor fusion by configuring the device to integrate track data using smoothed values. Furthermore, by applying the above configuration to the target tracking system including the target tracking device, or to the target tracking method, the same effects as described above can be achieved.

[0160] This embodiment further illustrates an example configuration including the following: [6] The second integration unit described above is The camera observation position, which is an observed value based on detection data from the imaging device, is replaced with the millimeter-wave observation position, which is an observed value based on detection data from the millimeter-wave radar device. The observed value and the track are then integrated using this replacement observed value. A target tracking device according to any one of claims 1 to 3. This disclosure further provides a target tracking device that enables high-precision target tracking using sensor fusion by correcting the observed values ​​integrated with the flight path to more suitable values. Furthermore, by applying the above configuration to the target tracking system including the target tracking device, or to the target tracking method, the same effects as described above can be achieved.

[0161] This embodiment further illustrates an example configuration including the following: [7] The second integration unit described above is Using the positions of surrounding moving objects included in the correlated observed values, the observed values ​​and the track are integrated by a linear Kalman filter. A target tracking device according to any one of [1], [2], [3], or [6], characterized by the above. This disclosure further provides a target tracking device that enables high-precision target tracking using sensor fusion by integrating the track using the position of each observed value. Furthermore, by applying the above configuration to the target tracking system including the target tracking device, or to the target tracking method, the same effects as described above can be achieved.

[0162] Here, we will describe the details of the hardware configuration required to realize the functions of this disclosure. Figure 9 shows a first example of a hardware configuration for realizing the functions of the configuration of this disclosure. Figure 10 shows a second example of a hardware configuration for realizing the functionality of the configuration described herein. Each of the target tracking devices 100 (100A, 100B, 100C) in this disclosure is implemented by hardware as shown in Figure 9 or Figure 10.

[0163] Each target tracking device 100 (100A, 100B, 100C) is composed of, for example, a processor 10001, a memory 10002, an input / output interface 10003, and a communication circuit 10004, as shown in Figure 9. Processor 10001 and memory 10002 are, for example, components installed in a computer. Memory 10002 stores a program that causes the computer to function as an observation value processing unit 110, a correlation processing unit 120, a tracking filter unit 130, an integration unit (first integration unit) 140, an identity determination unit 150, a track integration unit (second integration unit) 160, a position correction unit 170, a track management unit 180, a track output unit 190, and a control unit (not shown). By the processor 10001 reading and executing the program stored in memory 10002, the functions of the observation value processing unit 110, the correlation processing unit 120, the tracking filter unit 130, the integration unit (first integration unit) 140, the identity determination unit 150, the track integration unit (second integration unit) 160, the position correction unit 170, the track management unit 180, the track output unit 190, and the control unit (not shown) are realized. Furthermore, a storage unit (not shown) is realized by memory 10002 or other memory (not shown). Furthermore, a communication unit (not shown) is realized by the communication circuit 10004.

[0164] Processor 10001 uses, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microprocessor, a microcontroller, or a DSP (Digital Signal Processor). Memory 10002 may be a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), or flash memory; it may be a magnetic disk such as a hard disk or flexible disk; it may be an optical disk such as a CD (Compact Disc) or DVD (Digital Versatile Disc); or it may be a magneto-optical disk. The processor 10001, the memory 10002, or the communication circuit 10004 are connected in a state where they can transmit data to each other. Further, the processor 10001, the memory 10002, and the communication circuit 10004 are connected in a state where they can transmit data to and receive data from other hardware via the input / output interface 10003.

[0165] Alternatively, in the target tracking device 100 (100A, 100B, 100C), the functions of the observation value processing unit 110, the correlation processing unit 120, the tracking filter unit 130, the integration unit (first integration unit) 140, the identity determination unit 150, the track integration unit (second integration unit) 160, the position correction unit 170, the track management unit 180, the track output unit 190, and a control unit (not shown) may be realized by a dedicated processing circuit 20001 as shown in FIG. 10.

[0166] The processing circuit 20001 uses, for example, a single circuit, a composite circuit, a programmed processor, a parallel-programmed processor, an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), a SoC (System-on-a-Chip), or a system LSI (Large-Scale Integration), etc. Further, a storage unit (not shown) is realized by the memory 20002 or another memory (not shown). Memory 20002 may be a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), or flash memory; it may be a magnetic disk such as a hard disk or flexible disk; it may be an optical disk such as a CD (Compact Disc) or DVD (Digital Versatile Disc); or it may be a magneto-optical disk. Furthermore, a communication unit (not shown) is realized by the communication circuit 20004. The processing circuit 20001 and the memory 20002 or the communication circuit 20004 are connected in a way that allows them to transmit data to each other. Furthermore, the processing circuit 20001, the memory 20002, and the communication circuit 20004 are connected in a way that allows them to transmit data to other hardware via the input / output interface 20003. Furthermore, the functions of the observation value processing unit 110, correlation processing unit 120, tracking filter unit 130, integration unit (first integration unit) 140, identity determination unit 150, track integration unit (second integration unit) 160, position correction unit 170, track management unit 180, track output unit 190, and control unit (not shown) in the target tracking device 100 (100A, 100B, 100C) may be implemented by separate processing circuits, or they may be implemented together in a single processing circuit.

[0167] Alternatively, some functions of the target tracking device 100 (100A, 100B, 100C), including the observation value processing unit 110, correlation processing unit 120, tracking filter unit 130, integration unit (first integration unit) 140, identity determination unit 150, track integration unit (second integration unit) 160, position correction unit 170, track management unit 180, track output unit 190, and control unit (not shown), may be implemented by the processor 10001 and memory 10002, while the remaining functions are implemented by the processing circuit 20001.

[0168] Within the scope of this disclosure, it is possible to freely combine the embodiments, modify any component of each embodiment, or omit any component of each embodiment. For example, each embodiment may have all the functions of the respective embodiments, and each function may be configured to be switched and executed. [Industrial applicability]

[0169] This disclosure enables high-precision target tracking using sensor fusion, making it suitable for use in target tracking devices that track targets such as moving objects. [Explanation of Symbols]

[0170] 1(1A,1B,1C) Target tracking system, 11 First sensor, 12 Second sensor, 100(100A,100B,100C) Target tracking device, 110 Observation value processing unit, 120 Correlation processing unit, 130 Tracking filter unit, 140 Integration unit (First integration unit), 150 Identity determination unit, 160 Track integration unit (Second integration unit), 170 Position correction unit, 180 Track management unit, 190 Track output unit, 300 Output destination device, 10001 Processor, 10002 Memory, 10003 Input / output interface, 10004 Communication circuit, 20001 Processing circuit, 20002 Memory, 20003 Input / output interface, 20004 Communication circuit.

Claims

1. An observation processing unit that acquires the direction cosine for the direction of movement of a surrounding moving object for each observed value of the surrounding moving object detected based on the detection data of multiple sensors, A correlation processing unit that calculates the correlation between observed values ​​based on the direction cosine, A first integration unit integrates the observed values ​​that have been correlated by the correlation processing unit, An identity determination unit that determines whether the difference between the direction cosine of the observed value and the direction cosine of the stored track is within the range of the residual covariance matrix, A second integration unit that integrates the observed value and the track and outputs the result when the identity determination unit determines that the result is within the range of the residual covariance matrix, A target tracking device equipped with [a specific feature].

2. An observation processing unit that acquires the velocity vector and direction cosine with respect to the direction of movement of a surrounding moving object for each observed value of the surrounding moving object detected based on the detection data of multiple sensors, A correlation processing unit that calculates the correlation between observed values ​​based on the velocity vector and the direction cosine, A first integration unit integrates the observed values ​​that have been correlated by the correlation processing unit, An identity determination unit that determines whether the difference between the observed velocity vector and the stored track velocity vector, and the difference between the observed direction cosine and the stored track direction cosine, is within the range of the residual covariance matrix. A second integration unit that integrates the observed value and the track and outputs the result when the identity determination unit determines that the result is within the range of the residual covariance matrix, A target tracking device equipped with [a specific feature].

3. An observation value processing unit that acquires the width and depth of each surrounding moving object detected based on the detection data of the imaging device, A coarse gate is set based on the width and depth of the surrounding moving object, and for each coarse gate, a correlation processing unit is used to calculate the correlation between the observed values ​​based on the position of each observed value of the surrounding moving object detected based on the detection data of the millimeter-wave radar device. A first integration unit integrates the observed values ​​that have been correlated by the correlation processing unit, An identity determination unit that determines whether the difference between the position of the observed value and the position of the track is within the range of the residual covariance matrix, A second integration unit that integrates the observed value and the track and outputs the result when the identity determination unit determines that the result is within the range of the residual covariance matrix, A target tracking device equipped with [a specific feature].

4. The second integration unit described above is: Using the direction cosines of the observed values ​​correlated by the correlation processing unit, the observed values ​​and the track are integrated by an extended Kalman filter. A target tracking device according to any one of claims 1 to 3.

5. The first integration unit described above is By calculating the smoothed value of the correlated observed values, the observed values ​​are integrated. The second integration unit described above is: The observed values ​​and the track are integrated using the smoothed values. A target tracking device according to any one of claims 1 to 3.

6. The second integration unit described above is: The camera observation position, which is an observed value based on detection data from the imaging device, is replaced with the millimeter-wave observation position, which is an observed value based on detection data from the millimeter-wave radar device. The observed value and the track are then integrated using this replacement observed value. A target tracking device according to any one of claims 1 to 3.

7. The second integration unit described above is: Using the positions of surrounding moving objects included in the correlated observed values, the observed values ​​and the track are integrated by a linear Kalman filter. A target tracking device according to any one of claims 1 to 3.

8. The second integration unit described above is: Using the positions of surrounding moving objects included in the correlated observed values, the observed values ​​and the track are integrated by a linear Kalman filter. The target tracking device according to claim 6.

9. A target tracking method using a target tracking device, The observation value processing unit of the target tracking device obtains the direction cosine for the direction of movement of each of the observed values ​​of the surrounding moving object detected based on the detection data of each of the multiple sensors, The correlation processing unit of the target tracking device calculates the correlation between the observed values ​​based on the direction cosine, The first integration unit of the target tracking device integrates the correlated observed values, The identity determination unit of the target tracking device determines whether the difference between the direction cosine of the observed value and the direction cosine of the stored track is within the range of the residual covariance matrix. The second integration unit of the target tracking device integrates the observed value and the track and outputs the result when the identity determination unit determines that the value is within the range of the residual covariance matrix. A target tracking method characterized by the following:

10. A target tracking method using a target tracking device, The observation processing unit of the target tracking device acquires the velocity vector and the direction cosine with respect to the direction of movement of the surrounding moving object for each observed value of the surrounding moving object detected based on the detection data of each of the multiple sensors. The correlation processing unit of the target tracking device performs correlation calculations between observed values ​​based on the velocity vector and the direction cosine. The first integration unit of the target tracking device integrates the observed values ​​that have been correlated by the correlation processing unit, The identity determination unit of the target tracking device determines whether the difference between the observed velocity vector and the stored track velocity vector, and the difference between the observed direction cosine and the stored track direction cosine, are within the range of the residual covariance matrix. The second integration unit of the target tracking device integrates the observed value and the track and outputs the result when the identity determination unit determines that the value is within the range of the residual covariance matrix. A target tracking method characterized by the following:

11. A target tracking method using a target tracking device, The observation value processing unit of the target tracking device acquires the width and depth of each surrounding moving object detected based on the detection data of the imaging device. The correlation processing unit of the target tracking device sets a coarse gate based on the width and depth of the surrounding moving object, and for each coarse gate, it calculates the correlation between the observed values ​​based on the position of each observed value of the moving object detected based on the detection data of the millimeter-wave radar device. The first integration unit of the target tracking device integrates the observed values ​​that have been correlated by the correlation processing unit, The identity determination unit of the target tracking device determines whether the difference between the position of the observed value and the position of the track is within the range of the residual covariance matrix. The second integration unit of the target tracking device integrates the observed value and the track and outputs the result when the identity determination unit determines that the value is within the range of the residual covariance matrix. A target tracking method characterized by the following:

Images