Tracking device

Abstract

To reduce arithmetic load for tracking an object.SOLUTION: A tracking device 4 calculates first and second identical object probabilities on the basis of first and second detection information acquired from sensors 2, 3 and an object state that was estimated just before acquiring the first and second detection information. The tracking device 4 generates first and second post-observation type determination information indicating the object state estimated on the basis of the first and second detection information, the object state that was estimated just before the first and second detection information were acquired, and the first and second identical object probabilities. The tracking device 4 calculates the first and second identical object probabilities using first and second type likelihoods calculated on the basis of the first and second detection information, first and second position likelihoods, and first and second size likelihoods. The tracking device 4 calculates first and second similarities between the first and second type detection information and the second and first post-observation type determination information indicating the object state estimated just before the first and second type detection information were acquired with respect to first and second type detection information expressed by three values of the first and second detection information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a tracking device for tracking an object.

Background Art

[0002] Patent Document 1 describes an automatic driving system configured to receive detection information indicating detection results in a detection area common to a plurality of sensors from each of the plurality of sensors, and generate a plurality of hypotheses regarding an object existing in the common detection area based on the received detection information.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a tracking device that tracks an object based on detection information of a sensor, if generation of a plurality of hypotheses for tracking the object is continued, there is a problem that the number of hypotheses increases significantly over time and the computational load becomes large.

[0005] An object of the present disclosure is to reduce the computational load for tracking an object.

Means for Solving the Problems

[0006] One aspect of the present disclosure is a tracking device (4) configured to periodically acquire at least one piece of observation information indicating an observation result from at least one sensor (2, 3) configured to observe the surrounding state, and estimate and track an object state, which is a state of an object existing in the surroundings, based on the at least one piece of observation information, the tracking device (4) including a state update unit (S10 to S100).

[0007] The state update unit is configured to re-estimate the object state based on the acquired observation information each time it acquires at least one observation information from at least one sensor.

[0008] The state update unit comprises a probability calculation unit (S10-S30, S60-S80) and a state estimation unit (S40-S50, S90-S100). The probability calculation unit is configured to calculate the probability of identity, which indicates the probability that the acquired at least one observation is information obtained from an object corresponding to the previously estimated object state, based on at least one observation from at least one sensor and the object state estimated by the state update unit immediately before acquiring the at least one observation.

[0009] The state estimation unit is configured to re-estimate the object state by generating object state estimation information indicating the estimated object state based on at least one observation information from at least one sensor, the object state estimated by the state update unit immediately before acquiring at least one observation information, and the probability of identical objects calculated by the probability calculation unit.

[0010] The probability calculation unit includes a likelihood calculation unit (S10-S20, S60-S70). The likelihood calculation unit is configured to calculate at least one likelihood for each of at least one observational information, indicating the likelihood that it is information about an object corresponding to the object state estimated by the state update unit immediately before acquiring at least one observational information. The probability calculation unit then calculates the probability of identical objects using the at least one likelihood calculated by the likelihood calculation unit.

[0011] The likelihood calculation unit includes a similarity calculation unit (S10, S60). The similarity calculation unit is configured to calculate the similarity between a multi-value observation information, which is observation information represented by multiple values ​​from at least one set of observation information, and object state estimation information, which indicates the object state estimated by the state update unit immediately before acquiring the multi-value observation information.

[0012] The tracking device of this disclosure, configured in this manner, can convert multi-value observation information, represented by multiple values, into a similarity score represented by a single value, thereby suppressing the increase in hypotheses and enabling estimation of the object state. As a result, the tracking device of this disclosure can reduce the computational load required to track an object. [Brief explanation of the drawing]

[0013] [Figure 1] This is a diagram showing the configuration of the driver assistance system. [Figure 2] This is a flowchart showing the tracking process of the first embodiment. [Figure 3] This figure shows vectors A and B. [Figure 4] This is a diagram showing the likelihood function. [Figure 5] This diagram shows the method for calculating the first-class determination information. [Figure 6] This figure shows the method for calculating the species determination information after the first observation. [Figure 7] This graph shows the processing time for each frame. [Figure 8] This is a flowchart showing the tracking process of the second embodiment. [Modes for carrying out the invention]

[0014] [First Embodiment] A first embodiment of this disclosure is described below with reference to the drawings. The driver assistance system 1 of this embodiment is mounted on a vehicle and, as shown in Figure 1, comprises a camera sensor 2, a radar sensor 3, a tracking device 4, and a support execution unit 5. Hereinafter, the vehicle on which the driver assistance system 1 is mounted will be referred to as "the vehicle."

[0015] The camera sensor 2 is configured, for example, as a stereo camera capable of detecting the distance to an object, and based on the captured image, it detects the position (i.e., relative position to the vehicle), size, and type of an object in the image. In this embodiment, the object types are set to pedestrians, motorcycles, and automobiles. The camera sensor 2 generates first detection information indicating the detection result and transmits the generated first detection information to the tracking device 4. The first detection information includes first position detection information, first size detection information, and first type detection information. The first position detection information is information indicating the position of the object detected by the camera sensor 2. The first size detection information is information indicating the size of the object detected by the camera sensor 2. The first type detection information is information indicating the type of object detected by the camera sensor 2.

[0016] The radar sensor 3 transmits radar waves such as millimeter waves or microwaves and receives the reflected radar waves to detect the position (i.e., relative position to the vehicle), size, and type of an object. In this embodiment, the object types are set to pedestrians, motorcycles, and automobiles. The radar sensor 3 generates second detection information indicating the detection result and transmits the generated second detection information to the tracking device 4. The second detection information includes second position detection information, second size detection information, and second type detection information. The second position detection information is information indicating the position of the object detected by the radar sensor 3. The second size detection information is information indicating the size of the object detected by the radar sensor 3. The second type detection information is information indicating the type of object detected by the radar sensor 3.

[0017] The tracking device 4 is an electronic control device mainly composed of a microcomputer including a CPU 11, a ROM 12, a RAM 13, etc. Various functions of the microcomputer are realized by the CPU 11 executing a program stored in a non-transitory tangible recording medium. In this example, the ROM 12 corresponds to the non-transitory tangible recording medium storing the program. Further, by executing this program, a method corresponding to the program is executed. Note that part or all of the functions executed by the CPU 11 may be configured hardware-wise by one or more ICs or the like. Also, the number of microcomputers constituting the tracking device 4 may be one or plural.

[0018] Based on the first detection information generated by the camera sensor 2 and the second detection information generated by the radar sensor 3, the tracking device 4 tracks an object existing around the host vehicle and generates object information indicating the state of the tracked object.

[0019] The tracking device 4 uses a Kalman filter with the position of the object indicated by the first position detection information as an observed value of the position (hereinafter, the first position observed value) to calculate an estimated value of the position (hereinafter, the first position estimated value), a position standard deviation indicating the uncertainty of the estimated value of the position (hereinafter, the first position standard deviation), and a Kalman gain (hereinafter, the first position Kalman gain).

[0020] The tracking device 4 uses a Kalman filter with the size of the object indicated by the first size detection information as an observed value of the size (hereinafter, the first size observed value) to calculate an estimated value of the size (hereinafter, the first size estimated value), a size standard deviation indicating the uncertainty of the estimated value of the size (hereinafter, the first size standard deviation), and a Kalman gain (hereinafter, the first size Kalman gain).

[0021] The tracking device 4 uses a Kalman filter with the type of the object indicated by the first type detection information as an observed value of the type (hereinafter, the first type observed value) to calculate an estimated value of the type (hereinafter, the first type estimated value), a type standard deviation indicating the uncertainty of the estimated value of the type (hereinafter, the first type standard deviation), and a Kalman gain (hereinafter, the first type Kalman gain).

[0022] The tracking device 4 uses the position of the object indicated by the second position detection information as the observed position value (hereinafter referred to as the second position observation value) and uses a Kalman filter to calculate the estimated position value (hereinafter referred to as the second position estimate value), the position standard deviation (hereinafter referred to as the second position standard deviation) which indicates the uncertainty of the estimated position value, and the Kalman gain (hereinafter referred to as the second position Kalman gain).

[0023] The tracking device 4 uses the size of the object indicated by the second size detection information as the observed size (hereinafter referred to as the second size observed value) and uses a Kalman filter to calculate the estimated size (hereinafter referred to as the second size estimated value), the size standard deviation (hereinafter referred to as the second size standard deviation) which indicates the uncertainty of the estimated size, and the Kalman gain (hereinafter referred to as the second size Kalman gain).

[0024] The tracking device 4 uses the type of object indicated by the second type detection information as the observed type (hereinafter referred to as the second type observed value) and uses a Kalman filter to calculate the estimated type (hereinafter referred to as the second type estimated value), the type standard deviation (hereinafter referred to as the second type standard deviation) which indicates the uncertainty of the estimated type, and the Kalman gain (hereinafter referred to as the second type Kalman gain).

[0025] The support execution unit 5 includes, for example, an actuator, an audio device, and a display device. Based on the object information generated by the tracking device 4, the support execution unit 5 controls the behavior of its own vehicle and provides notifications to the driver of its own vehicle.

[0026] Next, the procedure for the tracking process executed by the CPU 11 of the tracking device 4 will be explained. The tracking process is a process that is repeatedly executed at predetermined processing cycles while the tracking device 4 is operating. The repetition period of the processing cycle is denoted as the execution period ΔT.

[0027] When the tracking process is executed, the CPU 11 first calculates the first similarity score s1 in S10, as shown in Figure 2. The first similarity score s1 is calculated using the first type detection information contained in the most recent first detection information transmitted from the camera sensor 2 and the second post-observation type determination information, described later, which was calculated in S100 of the tracking process one cycle prior to the execution cycle ΔT. Note that in S10 of the initial tracking process executed immediately after the tracking device 4 is started, the first similarity score s1 is not calculated because the second post-observation type determination information from the tracking process one cycle prior does not exist.

[0028] The first type detection information is represented by a vector showing the detection probability of pedestrians, motorcycles, and automobiles, respectively. The detection probability of a pedestrian is the probability that the object detected by camera sensor 2 is a pedestrian. Similarly, the detection probability of a motorcycle is the probability that the object detected by camera sensor 2 is a motorcycle. The detection probability of an automobile is the probability that the object detected by camera sensor 2 is an automobile. For example, if the first type information is [0.1,0.2,0.7], the detection probabilities of pedestrians, motorcycles, and automobiles are 10%, 20%, and 70%, respectively. The first type information is generated such that the sum of the detection probabilities of pedestrians, motorcycles, and automobiles equals 100%.

[0029] Furthermore, the second observation type determination information, like the first type detection information, is represented by vectors showing the detection probabilities for pedestrians, motorcycles, and automobiles, respectively. As shown in Figure 3, the CPU 11 calculates the first similarity s1 using equation (1), where vector A = [A1, A2, A3] represents the first type detection information and vector B = [B1, B2, B3] represents the second type determination information after observation. If vector A and vector B are identical, the first similarity s1 is 1.

[0030]

number

[0031] Next, as shown in Figure 2, the CPU 11 calculates the first type likelihood q11 in S20. The first type likelihood q11 is the degree of likelihood that the object detected by the camera sensor 2 (i.e., the object corresponding to the first type detection information) and the predicted object (i.e., the object corresponding to the second observation-based type determination information) are the same object.

[0032] The first type likelihood q11 is calculated using the first similarity s1, the type determination accuracy of the camera sensor 2, and the prediction type determination accuracy. Note that in S20 of the initial tracking process executed immediately after the tracking device 4 is started, the first type likelihood q11 is not calculated because the first similarity s1 from the tracking process one cycle prior does not exist.

[0033] The type determination accuracy of camera sensor 2 is a value between 0 and 1, and is a preset fixed value (for example, 0.2). A smaller value indicates better accuracy.

[0034] The accuracy of the prediction's type determination is the degree to which the type indicated by the second observation post-type determination information matches the type of the detected object. Specifically, the prediction's type determination accuracy is the standard deviation calculated using multiple second observation post-type determination information calculated in S100, described below.

[0035] The CPU 11 calculates the first type likelihood q11 by substituting the first similarity s1, the type determination accuracy of the camera sensor 2, and the prediction type determination accuracy into a pre-configured first type likelihood function. The first type likelihood function is a function for calculating the first type likelihood q11 using the first similarity s1, the type determination accuracy of the camera sensor 2, and the prediction type determination accuracy as variables. Alternatively, the first type likelihood q11 may be calculated by referring to a first type likelihood table in which the correspondence between the first similarity s1, the type determination accuracy of the camera sensor 2, the prediction type determination accuracy, and the first type likelihood q11 is defined.

[0036] Graph G1 in Figure 4 shows the first type likelihood function. However, for the sake of simplicity in the illustration, the first type likelihood function shown in Graph G1 only includes the first similarity variable, s1. Next, as shown in Figure 2, the CPU 11 calculates the first identical object probability r1 in S30. The first identical object probability r1 is the probability that the object detected by the camera sensor 2 (i.e., the object corresponding to the first type detection information) and the predicted object (i.e., the object corresponding to the second post-observation type determination information) are the same object.

[0037] The first identical object probability r1 is calculated using the first type likelihood q11, the first position likelihood q12, and the first size likelihood q13. Note that in S30 of the initial tracking process executed immediately after the tracking device 4 is started, the first type likelihood q11 from the tracking process one cycle prior does not exist, so the first identical object probability r1 is not calculated.

[0038] The first position likelihood q12 is the degree of plausibility of the first position observation. The CPU11 calculates the first position likelihood q12 by substituting the first position observation into the first position likelihood function, which is set based on the first position estimate and the first position standard deviation. The first position likelihood function is a function for calculating the first position likelihood q12 with the first position observation as the variable, as shown in graph G2 in Figure 4. The first position likelihood function follows a normal distribution with the first position estimate as the mean and the square of the first position standard deviation as the variance.

[0039] The first size likelihood q13 is the degree of likelihood of the first size observation. The CPU11 calculates the first size likelihood q13 by substituting the first size observation into the first size likelihood function, which is set based on the first size estimate and the first size standard deviation. The first size likelihood function is a function for calculating the first size likelihood q13 with the first size observation as the variable, as shown in graph G3 in Figure 4. The first size likelihood function exhibits a normal distribution with the first size estimate as the mean and the square of the first size standard deviation as the variance.

[0040] The CPU 11 then calculates the average of the first type likelihood q11, the first position likelihood q12, and the first size likelihood q13. Furthermore, it normalizes this average so that the maximum value of the average is 1, and the result is defined as the first identical object probability r1.

[0041] Next, as shown in Figure 2, CPU 11 calculates the first type determination information in S40. Specifically, as shown in Figure 5, CPU 11 calculates the first type determination information using equation (2), with vector A' representing the first type determination information and K1 being the first type Kalman gain.

[0042]

number

[0043] Next, as shown in Figure 2, the CPU 11 calculates the first post-observation classification information and the standard deviation of the first post-observation classification information in S50, and stores the calculated first post-observation classification information and standard deviation in the RAM 13.

[0044] Specifically, as shown in Figure 6, the CPU 11 calculates the first post-observation classification information using equation (3), with vector A'' representing the first post-observation classification information. Furthermore, the CPU 11 calculates the standard deviation of the first post-observation classification information using multiple pieces of first post-observation classification information stored in RAM 13.

[0045]

number

[0046] Next, CPU 11 calculates the second similarity score s2 in S60, as shown in Figure 2. The second similarity score s2 is calculated using the second type detection information included in the most recent second detection information transmitted from the radar sensor 3 and the most recent first observation post-type determination information calculated in S50.

[0047] Let vector C = [C1, C2, C3] represent the second type detection information, and vector D = [D1, D2, D3] represent the first observation-based type determination information. CPU 11 calculates the second similarity s2 using equation (4). If vector C and vector D are identical, the second similarity s2 is 1.

[0048]

number

[0049] Next, in S70, the CPU 11 calculates the second type likelihood q21. The second type likelihood q21 is the degree of likelihood that the object detected by the radar sensor 3 (i.e., the object corresponding to the second type detection information) and the predicted object (i.e., the object corresponding to the first observation type determination information) are the same object.

[0050] The second type likelihood q21 is calculated using the second similarity s2, the type determination accuracy of the radar sensor 3, and the type determination accuracy of the prediction. The type determination accuracy of radar sensor 3 is a value between 0 and 1, and is a preset fixed value (for example, 0.4).

[0051] The accuracy of the prediction's type determination is the degree to which the type indicated by the first post-observation type determination information matches the type of the detected object. Specifically, the prediction's type determination accuracy is the standard deviation calculated using multiple first post-observation type determination information calculated in S50.

[0052] The CPU 11 calculates the second type likelihood q21 by substituting the second similarity s2, the type determination accuracy of the radar sensor 3, and the prediction type determination accuracy into a pre-set second type likelihood function. The second type likelihood function is a function for calculating the second type likelihood q21 using the second similarity s2, the type determination accuracy of the radar sensor 3, and the prediction type determination accuracy as variables. Alternatively, the second type likelihood q21 may be calculated by referring to a second type likelihood table in which the correspondence between the second similarity s2, the type determination accuracy of the radar sensor 3, the prediction type determination accuracy, and the second type likelihood q21 is defined.

[0053] Next, CPU 11 calculates the second identical object probability r2 in S80. The second identical object probability r2 is the probability that the object detected by radar sensor 3 (i.e., the object corresponding to the second type detection information) and the predicted object (i.e., the object corresponding to the first observation type determination information) are the same object.

[0054] The second identical probability r² is calculated using the second type likelihood q²1, the second position likelihood q²2, and the second size likelihood q²3. The second position likelihood q22 represents the degree of plausibility of the second position observation. The CPU11 calculates the second position likelihood q22 by substituting the second position observation into the second position likelihood function, which is set based on the second position estimate and the second position standard deviation. The second position likelihood function is a function for calculating the second position likelihood q22 with the second position observation as the variable. The second position likelihood function exhibits a normal distribution with the second position estimate as the mean and the square of the second position standard deviation as the variance.

[0055] The second size likelihood q23 is the degree of likelihood of the second size observation. The CPU11 calculates the second size likelihood q23 by substituting the second size observation into the second size likelihood function, which is set based on the second size estimate and the second size standard deviation. The second size likelihood function is a function for calculating the second size likelihood q23 with the second size observation as the variable. The second size likelihood function exhibits a normal distribution with the second size estimate as the mean and the square of the second size standard deviation as the variance.

[0056] The CPU 11 then calculates the average of the second type likelihood q21, the second position likelihood q22, and the second size likelihood q23, and further normalizes this average so that the maximum value of the average is 1, and this value is defined as the second identical object probability r2.

[0057] Next, CPU 11 calculates the second type determination information in S90. Specifically, CPU 11 uses vector C' to represent the second type determination information and K2 to represent the second type Kalman gain, and calculates the second type determination information using equation (5).

[0058]

number

[0059] Next, in S100, the CPU 11 calculates the second observation type determination information and the standard deviation of the second observation type determination information, and stores the calculated second observation type determination information and standard deviation in the RAM 13.

[0060] Specifically, the CPU 11 calculates the second post-observation classification information using equation (6), with vector C'' representing the second post-observation classification information. Furthermore, the CPU 11 calculates the standard deviation of the second post-observation classification information using the multiple second post-observation classification information stored in RAM 13.

[0061]

number

[0062] Furthermore, the tracking device 4 determines whether the object detected by the camera sensor 2 and radar sensor 3 is a pedestrian, a motorcycle, or an automobile, based on the second observation type determination information.

[0063] Figure 7 is a graph comparing the time required to perform the object state estimation calculation in each frame between the method of this disclosure and the conventional method. Each frame corresponds to a process that is repeatedly executed each time the execution period ΔT described above has elapsed. The conventional method performs the object state estimation calculation using a PMB filter. PMB stands for Poisson Multi-Bernoulli.

[0064] As shown in Figure 7, the maximum computation time in the conventional method is 47 ms, while the maximum computation time in the method of this disclosure is 2.7 ms, demonstrating that the computation time can be reduced by the method of this disclosure.

[0065] The tracking device 4 configured in this way periodically acquires first and second detection information indicating the detection results from the camera sensor 2 and radar sensor 3, which are configured to observe the surrounding conditions, and estimates and tracks the state of objects present in the surroundings (hereinafter referred to as "object state") based on the first and second detection information.

[0066] The tracking device 4 is configured to re-estimate the object state based on the acquired first and second detection information each time it acquires first and second detection information from the camera sensor 2 and radar sensor 3. The tracking device 4 is configured to calculate a first identical object probability r1, which indicates the probability that the acquired first detection information is information obtained from an object corresponding to the previously estimated object state, based on the first detection information from the camera sensor 2 and the object state estimated immediately before acquiring the first detection information (i.e., the second post-observation type determination information).

[0067] The tracking device 4 is configured to calculate a second identical object probability r2, which indicates the probability that the acquired second detection information is information obtained from an object corresponding to the previously estimated object state, based on the second detection information from the radar sensor 3 and the object state estimated immediately before acquiring the second detection information (i.e., the first post-observation type determination information).

[0068] The tracking device 4 is configured to re-estimate the object state by generating first post-observation classification information indicating the estimated object state, based on first detection information from the camera sensor 2, the object state estimated immediately before acquiring the first detection information (i.e., second post-observation classification information), and the calculated first identical object probability r1.

[0069] The tracking device 4 is configured to re-estimate the object state by generating second post-observation classification information indicating the estimated object state, based on the second detection information from the radar sensor 3, the object state estimated immediately before acquiring the second detection information (i.e., first post-observation classification information), and the calculated second identical object probability r2.

[0070] The tracking device 4 calculates the first identical object probabilities r1 and r2 for each of the first and second detection information using the first type likelihoods q11 and q21, the first and second position likelihoods q12 and q22, and the first and second size likelihoods q13 and q23, which indicate the likelihood that the information corresponds to the object state estimated immediately before acquiring the first and second detection information. Then, the tracking device 4 calculates the first and second identical object probabilities r1 and r2 using the calculated first and second type likelihoods q11 and q21, the first and second position likelihoods q12 and q22, and the first and second size likelihoods q13 and q23.

[0071] The tracking device 4 is configured to calculate the first and second similarity s1 and s2 between the first and second type detection information, which is represented by three values ​​from the first and second detection information, and the second and first post-observation type determination information, which indicates the state of the object estimated immediately before acquiring the first and second type detection information.

[0072] Such a tracking device 4 can convert first and second type detection information, which is represented by multiple values, into first and second similarity s1 and s2, which are represented by a single value, thereby suppressing the increase in hypotheses and estimating the state of the object. As a result, the tracking device 4 can reduce the computational load required to track the object.

[0073] Furthermore, the tracking device 4 generates first and second post-observation type determination information by integrating a first hypothetical object state (i.e., vectors A', C'), which is the object state when it is assumed that the first type detection information is information about an object corresponding to the object state estimated immediately before acquiring the first type detection information, and a second hypothetical object state (i.e., vectors B, D), which is the object state when it is assumed that the first type detection information is not information about an object corresponding to the object state estimated immediately before acquiring the first type detection information, using first and second identical object probabilities r1, r2.

[0074] Since such a tracking device 4 estimates the object state by considering two types of states, a first hypothetical object state and a second hypothetical object state, it can improve the robustness of object state estimation.

[0075] Furthermore, the tracking device 4 uses the first detection information obtained from the camera sensor 2 to estimate the object state as the first object state. Then, the tracking device 4 uses the second detection information obtained from the radar sensor 3 and the estimated first object state to estimate the object state as the second object state.

[0076] Such a tracking device 4 can estimate the state of an object by integrating detection information from two sensors. In the embodiments described above, the camera sensor 2 and radar sensor 3 correspond to sensors, the first and second detection information corresponds to observation information, S10 to S100 correspond to processing as a state update unit, S10 to S30 and S60 to S80 correspond to processing as a probability calculation unit, and the identical object probabilities correspond to the first and second identical object probabilities r1 and r2.

[0077] Furthermore, S40-S50 and S90-S100 correspond to processing as a state estimation unit, the type determination information after the first and second observations corresponds to object state estimation information, S10-S20 and S60-S70 correspond to processing as a likelihood calculation unit, and likelihoods q11, q21, q12, q22, q13, and q23 correspond to likelihoods.

[0078] Furthermore, S10 and S60 correspond to processing as a similarity calculation unit, and the first and second type detection information corresponds to multiple value observation information. Furthermore, camera sensor 2 corresponds to the first sensor, radar sensor 3 corresponds to the second sensor, S10 to S50 correspond to processing as the first estimation unit, and S60 to S100 correspond to processing as the second estimation unit.

[0079] [Second Embodiment] A second embodiment of this disclosure is described below with reference to the drawings. Note that the second embodiment will describe parts that differ from the first embodiment. Common components will be denoted by the same reference numerals.

[0080] The driving assistance system 1 of the second embodiment differs from the first embodiment in that the tracking process has been modified. The tracking process in the second embodiment differs from that of the first embodiment in that the processes S22, S24, S26, S72, S74, and S76 are added, as shown in Figure 8.

[0081] In other words, when the processing in S20 is completed, the CPU 11 calculates the first Mahalanobis distance in S22 between vector A used in S10 and multiple second observation type determination information stored in RAM 13 (i.e., vectors B and C'').

[0082] Specifically, CPU 11 calculates the mean vector and variance-covariance matrix of multiple vectors B (or vector C'), and calculates the first Mahalanobis distance by substituting vector A and the mean vector and variance-covariance matrix of multiple vectors B into a well-known formula for calculating the Mahalanobis distance.

[0083] Then, in S24, CPU 11 determines whether the first Mahalanobis distance calculated in S22 is below a predetermined threshold. If the first Mahalanobis distance is below the threshold, CPU 11 proceeds to S30. On the other hand, if the first Mahalanobis distance exceeds the threshold, CPU 11, in S26, changes the second post-observation classification information (i.e., vector B) used in S10 to the first post-observation classification information (i.e., vector A'') and proceeds to S60.

[0084] Furthermore, once the processing in S70 is complete, in S72 the CPU 11 calculates the second Mahalanobis distance between the vector C used in S60 and the multiple first observation type determination information (i.e., vector A'') stored in RAM 13.

[0085] Specifically, CPU 11 calculates the mean vector and covariance matrix of multiple vectors A'', and calculates the second Mahalanobis distance by substituting vector C and the mean vector and covariance matrix of multiple vectors A'' into a well-known formula for calculating the Mahalanobis distance.

[0086] Then, in S74, CPU 11 determines whether the second Mahalanobis distance calculated in S72 is below a predetermined threshold. If the second Mahalanobis distance is below the threshold, CPU 11 proceeds to S80. On the other hand, if the second Mahalanobis distance exceeds the threshold, CPU 11, in S76, uses the first post-observation classification information (i.e., vector D) used in S60 as the second post-observation classification information (i.e., vector C'') and terminates the tracking process.

[0087] The tracking device 4 configured in this way calculates the first Mahalanobis distance between the first type determination information (i.e., vector A) acquired from the camera sensor 2 and the generated second post-observation type determination information (i.e., vector B). If the first Mahalanobis distance is greater than a preset threshold, the tracking device 4 determines that the object corresponding to the second post-observation type determination information was not observed by the camera sensor 2.

[0088] The tracking device 4 also calculates the second Mahalanobis distance between the first classification information (i.e., vector C) acquired from the radar sensor 3 and the generated first post-observation classification information (i.e., vector A''). If the second Mahalanobis distance is greater than a preset threshold, the tracking device 4 determines that the object corresponding to the first post-observation classification information was not observed by the radar sensor 3.

[0089] Such a tracking device 4 can omit calculations using detection results with large Mahalanobis distances, thereby suppressing a decrease in the accuracy of estimating the object state and speeding up the calculation of estimating the object state.

[0090] In the embodiments described above, S22 and S72 correspond to processing as a Mahalanobis distance calculation unit, the first and second type detection information corresponds to observation information, the first and second Mahalanobis distances correspond to Mahalanobis distances, and S24 and S74 correspond to processing as an observation judgment unit.

[0091] Although one embodiment of the present disclosure has been described above, the present disclosure is not limited to the above embodiment and can be implemented in various modified forms. [Example 1] In the above embodiment, the object state is estimated based on first and second detection information acquired from the camera sensor 2 and the radar sensor 3. However, the tracking device 4 may also estimate the object state based on the detection information from either the camera sensor 2 or the radar sensor 3. Furthermore, if the tracking device 4 acquires detection information from three or more sensors, it may also estimate the object state based on the detection information acquired from three or more sensors.

[0092] [Differentiation 2] In the above embodiment, the tracking device 4 was shown to estimate the state of an object. However, the tracking device 4 may also calculate the probability that an object exists (hereinafter referred to as the object existence probability). This allows the tracking device 4 to update parameters related to the occurrence / disappearance of an object (i.e., the object existence probability) during object tracking. The tracking device 4 calculates the object existence probability using, for example, a PMBM filter or a PMB filter. PMBM stands for Poisson Multi-Bernoulli Mixture.

[0093] [Difference 3] In the above embodiment, a method was shown in which the first identical object probabilities r1 and r2 are calculated using the first type likelihoods q11 and q21, the first and second position likelihoods q12 and q22, and the first and second size likelihoods q13 and q23. However, the tracking device 4 may also calculate the first identical object probabilities r1 and r2 using random finite set theory, such as a PHD filter, LMB filter, PMBM filter, or PMB filter. By using random finite set theory, such a tracking device 4 can improve the robustness of object estimation. PHD stands for Probability Hypothesis Density. LMB stands for Labeled Multi-Bernoulli.

[0094] [Differentiation Example 4] In the above embodiment, a configuration was shown in which the first and second similarity scores s1 and s2 were calculated using first and second type detection information. However, it is also possible to calculate the similarity using multidimensional feature information of objects obtained particularly through image learning as detection information. Such a tracking device 4 can estimate the state of an object using information that does not have a physical interpretation for each individual numerical value, such as feature information output by AI. AI stands for Artificial Intelligence.

[0095] The tracking device 4 and its method described herein may be implemented by a dedicated computer provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. Alternatively, the tracking device 4 and its method described herein may be implemented by a dedicated computer provided by configuring a processor by one or more dedicated hardware logic circuits. Alternatively, the tracking device 4 and its method described herein may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to perform one or more functions and a processor configured by one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium. The method for implementing the functions of each part included in the tracking device 4 does not necessarily have to include software, and all of its functions may be implemented using one or more hardware components.

[0096] Multiple functions of one component in the above embodiment may be realized by multiple components, or one function of one component may be realized by multiple components. Furthermore, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Also, some parts of the configuration of the above embodiment may be omitted. Furthermore, at least some parts of the configuration of the above embodiment may be added to or replaced with the configuration of other above embodiments.

[0097] In addition to the tracking device 4 described above, this disclosure can also be implemented in various forms, such as a system comprising the tracking device 4, a program for causing a computer to function as the tracking device 4, a non-transitional physical recording medium such as a semiconductor memory on which this program is recorded, and a tracking method. [Explanation of Symbols]

[0098] 2...Camera sensor, 3...Radar sensor, 4...Tracking device

Claims

1. A tracking device (4) periodically acquires at least one piece of observational information indicating the observation result from at least one sensor (2, 3) configured to observe the surrounding conditions, and estimates and tracks the state of an object, which is the state of an object present in the surroundings, based on the at least one piece of observational information, The system includes a state update unit (S10 to S100) configured to re-estimate the state of the object based on the acquired observation information each time at least one observation information is acquired from the at least one sensor, The aforementioned state update unit, A probability calculation unit (S10 to S30, S60 to S80) is configured to calculate the probability that the acquired at least one observation information is information obtained from the object corresponding to the immediately estimated object state, based on the at least one observation information from the at least one sensor and the object state estimated by the state update unit immediately before acquiring the at least one observation information, The system includes a state estimation unit (S40-S50, S90-S100) configured to re-estimate the object state by generating object state estimation information indicating the estimated object state based on the at least one observation information from the at least one sensor, the object state estimated by the state update unit immediately before acquiring the at least one observation information, and the probability of identical objects calculated by the probability calculation unit, The probability calculation unit, The system includes a likelihood calculation unit (S10-S20, S60-S70) configured to calculate at least one likelihood for each of the at least one observational pieces of information that indicates the likelihood that the information corresponds to the object state estimated by the state update unit immediately before acquiring the at least one observational piece of information, and calculates the probability of identical objects using the at least one likelihood calculated by the likelihood calculation unit. The likelihood calculation unit is a tracking device comprising a similarity calculation unit (S10, S60) configured to calculate the similarity between the multi-value observation information, which is observation information represented by multiple values ​​from the at least one piece of observation information, and the object state estimation information, which indicates the object state estimated by the state update unit immediately before acquiring the multi-value observation information.

2. A tracking device according to claim 1, The state estimation unit, A tracking device that generates object state estimation information by integrating, using the same-object probability, a first hypothetical object state which is the object state when it is assumed that the at least one piece of observational information is information about the object corresponding to the object state estimated by the state update unit immediately before acquiring the at least one piece of observational information, and a second hypothetical object state which is the object state when it is assumed that the at least one piece of observational information is not information about the object corresponding to the object state estimated by the state update unit immediately before acquiring the at least one piece of observational information.

3. A tracking device according to claim 1 or claim 2, The aforementioned at least one sensor is a plurality of sensors, The plurality of sensors include a first sensor and a second sensor that are different from each other. The aforementioned state update unit, A first estimation unit (S10 to S50) estimates the object state as a first object state using the at least one observation information obtained from the first sensor, A second estimation unit (S60 to S100) estimates the object state as a second object state using the at least one observation information obtained from the second sensor and the first object state estimated by the first estimation unit. A tracking device equipped with the following features.

4. A tracking device according to claim 1 or claim 2, The state estimation unit is a tracking device configured to calculate the probability of the object's existence, in addition to the object's state.

5. A tracking device according to claim 1 or claim 2, The state update unit is a tracking device that updates the state of the object using random finite set theory, such as a PHD filter, LMB filter, PMBM filter, or PMB filter.

6. A tracking device according to claim 1 or claim 2, The aforementioned multi-value observation information is a tracking device that provides multidimensional feature information of the object.

7. A tracking device according to claim 1 or claim 2, A Mahalanobis distance calculation unit (S22, S72) is configured to calculate the Mahalanobis distance between the at least one observation information obtained from the at least one sensor and the object state estimation information generated by the state estimation unit, If the Mahalanobis distance is greater than a preset threshold, the observation determination unit (S24, S74) is configured to determine that the object corresponding to the object state estimation information was not observed by the at least one sensor. A tracking device equipped with the following features.

Images