Object tracking device and object tracking method

Abstract

To provide an object tracking device and an object tracking method that can accurately track an object without increasing a load of calculation.SOLUTION: An object tracking device (20) comprises: an input interface (21) that acquires sensor data; a processor (23) that detects an object to be detected from the sensor data and tracks the object to be detected by using Kalman filters associated with the object to be detected and an observed value; and an output interface (24) that outputs a result of detection of the object to be detected. The processor (23) executes first processing of selecting, from the plurality of Kalman filters associated with the same object to be detected or observed value, ones with a low probability as exclusion candidate Kalman filters that can be excluded from the association, and second processing of excluding the exclusion candidate Kalman filters satisfying an initialization condition from the association and initializing the excluded filters.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an object tracking device and an object tracking method. [Background technology]

[0002] There are known techniques for detecting surrounding objects, tracking the detected objects, and predicting their movements. For example, Patent Document 1 discloses a device that processes video signals output from an on-board camera that captures video around a vehicle, detects the presence or absence of approaching vehicles and pedestrians, and displays the approaching vehicles and pedestrians by adding square frame marks. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-321494 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional technology, vehicles detected as individual on an image may actually be the same vehicle, or conversely, vehicles detected as a single vehicle may actually be separate vehicles, so there is a need for further improvements in the accuracy of object tracking.

[0005] In view of the above, an object of the present disclosure is to provide an object tracking device and an object tracking method that can track an object with high accuracy without increasing the computational load. [Means for solving the problem]

[0006] An object tracking device according to an embodiment includes: an input interface for acquiring sensor data; a processor that detects a detection target from the sensor data and tracks the detection target using a Kalman filter in which the detection target and an observation value are associated with each other; an output interface for outputting a detection result of the detection object; The processor, a first process of selecting, from among a plurality of Kalman filters associated with the same detection target or observation value, a filter having a low degree of certainty as a candidate Kalman filter that may be excluded from the association; A second process is executed to exclude the candidate Kalman filter that satisfies an initialization condition from the association and initialize the Kalman filter.

[0007] An object tracking method according to an embodiment includes: acquiring sensor data; Detecting a detection target from the sensor data, and tracking the detection target using a Kalman filter in which the detection target and the observation value are associated with each other; and outputting a detection result of the detection target. The tracking of the detection target includes: a first process of selecting, from among a plurality of Kalman filters associated with the same detection target or observation value, a filter having a low degree of certainty as a candidate Kalman filter that may be excluded from the association; and a second process of initializing the candidate Kalman filter for exclusion that satisfies an initialization condition from the correspondence. Effect of the Invention

[0008] According to the embodiments of the present disclosure, it is possible to provide an object tracking device and an object tracking method that can track an object with high accuracy without increasing the computational load. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of an object tracking system including an object tracking device according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing a vehicle equipped with the object tracking system of FIG. 1 and a detection target. [Diagram 3]FIG. 3 is a flowchart showing an example of a process for tracking an image of an object on a moving image. [Figure 4] FIG. 4 is a diagram showing an example of an image of an object on a moving image. [Diagram 5] FIG. 5 is a diagram for explaining the relationship between an object in real space, an image of the object in a moving image, and a mass point in virtual space. [Figure 6] FIG. 6 is a diagram showing an example of movement of a mass point in a virtual space. [Figure 7] FIG. 7 is a diagram for explaining data association. [Figure 8] FIG. 8 is a diagram illustrating a hierarchical structure of tracking object ID management. [Figure 9] FIG. 9 is a diagram showing error ellipses of overlapping Kalman filters associated with the same observation point. [Figure 10] FIG. 10 is a diagram showing error ellipses of the overlapping Kalman filters associated with the same detection target. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The drawings used in the following description are schematic. The dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0011] Fig. 1 is a block diagram showing a schematic configuration of an object tracking system 1. An object tracking device 20 according to an embodiment of the present disclosure is included in the object tracking system 1. In this embodiment, the object tracking system 1 includes an imaging device 10, the object tracking device 20, and a display 30. In this embodiment, the object tracking system 1 is mounted on a vehicle 100, which is an example of a moving body, as illustrated in Fig. 2.

[0012] The object tracking device 20 according to this embodiment acquires a moving image from the imaging device 10 as sensor data. That is, in this embodiment, the sensor used to detect the detection target is the imaging element 12 that captures visible light provided in the imaging device 10. However, the object tracking system 1 is not limited to the configuration shown in FIG. 1. The object tracking system 1 can be provided with a device different from the imaging device 10 as long as it detects the detection target. As another example, the object tracking system 1 may be configured to include a measuring device that measures the distance to the detection target from the reflected wave of the irradiated laser light instead of the imaging device 10. As another example, the object tracking system 1 may be configured to include a detection device having a millimeter wave sensor instead of the imaging device 10. Also, as another example, the object tracking system 1 may be configured to include an imaging device 10 that includes an imaging element 12 that captures light other than the visible light region.

[0013] In this embodiment, the object tracking system 1 is mounted on a moving body, and detects objects 40 (see FIG. 2) around the moving moving body. However, the object tracking system 1 is not limited to being mounted on a moving body. As another example, the object tracking system 1 may be used in a facility such as a factory, and detect employees, transport robots, and products. As another example, the object tracking system 1 may be used in an elderly welfare facility, and detect elderly people and staff in the room. The object tracking system 1 may not only track objects for the safety of driving or activities, but also track objects for the efficiency of work, quality control, or productivity improvement in agricultural and industrial sites. Here, in the present disclosure, objects that are detected by the object tracking device 20 include not only objects such as moving bodies, but also people.

[0014] As shown in FIG. 2, in this embodiment, the x-axis direction of the coordinates of the real space is the width direction of the vehicle 100 on which the imaging device 10 is installed. The positive y-axis direction is the direction in which the vehicle 100 moves backward. The x-axis and y-axis directions are parallel to the road surface on which the vehicle 100 is located. The z-axis direction is perpendicular to the road surface. The z-axis direction can be called the vertical direction. The x-axis, y-axis, and z-axis directions are mutually orthogonal. The way in which the x-axis, y-axis, and z-axis directions are taken is not limited to this. The x-axis, y-axis, and z-axis directions can be interchanged with each other.

[0015] The imaging device 10 includes an imaging optical system 11, an imaging element 12, and a processor 13.

[0016] The imaging device 10 may be installed at various positions on the vehicle 100. The imaging device 10 includes, but is not limited to, a front camera, a left side camera, a right side camera, and a rear camera. The front camera, the left side camera, the right side camera, and the rear camera are installed on the vehicle 100 so as to be able to capture images of the surrounding areas in front of, on the left side of, on the right side of, and behind the vehicle 100, respectively. In an embodiment described below as an example, as shown in FIG. 2, the imaging device 10 is attached to the vehicle 100 with its optical axis direction facing downward from the horizontal direction so as to be able to capture images of the rear of the vehicle 100.

[0017] The imaging optical system 11 may include one or more lenses. The imaging element 12 may include a charge-coupled device image sensor (CCD image sensor) or a complementary MOS image sensor (CMOS image sensor).

[0018] The imaging element 12 converts an image of an object (subject image) formed on the imaging surface of the imaging element 12 by the imaging optical system 11 into an electrical signal. The imaging element 12 can capture moving images at a predetermined frame rate. A frame is each still image that constitutes a moving image. The number of images that can be captured per second is called the frame rate. The frame rate may be, for example, 60 fps (frames per second) or 30 fps.

[0019] The processor 13 controls the entire imaging device 10 and performs various image processing on the moving images output from the imaging element 12. The image processing performed by the processor 13 may include any processing such as distortion correction, brightness adjustment, contrast adjustment, gamma correction, etc.

[0020] The processor 13 may be comprised of one or more processors. The processor 13 includes one or more circuits or units configured to perform one or more data computing procedures or processes, for example, by executing instructions stored in associated memory. The processor 13 includes one or more processors, microprocessors, microcontrollers, application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), any combination of these devices or configurations, or other known devices or configurations.

[0021] The object tracking device 20 includes an input interface 21, a memory unit 22, a processor 23, and an output interface 24.

[0022] The input interface 21 is configured to be able to communicate with the imaging device 10 via a wired or wireless communication means. The input interface 21 acquires moving images from the imaging device 10 as sensor data. The input interface 21 may correspond to a transmission method of an image signal transmitted by the imaging device 10. The input interface 21 can be referred to as an input unit or an acquisition unit. The imaging device 10 and the input interface 21 may be connected via an in-vehicle communication network such as a control area network (CAN).

[0023] The storage unit 22 is a storage device that stores data and programs necessary for processing performed by the processor 23. For example, the storage unit 22 temporarily stores moving images acquired from the imaging device 10. For example, the storage unit 22 stores data generated by processing performed by the processor 23. The storage unit 22 may be configured using, for example, one or more of a semiconductor memory, a magnetic memory, an optical memory, and the like. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk and a magnetic tape. The optical memory may include, for example, a compact disc (CD), a digital versatile disc (DVD), and a Blu-ray (registered trademark) disc (BD).

[0024] The processor 23 controls the entire object tracking device 20. The processor 23 recognizes an image of an object included in a moving image acquired via the input interface 21. The processor 23 performs mapping conversion of the coordinates of the recognized image of the object into the coordinates of the object 40 in a virtual space 46 (see FIG. 6), and tracks the position and velocity of a mass point 45 (see FIG. 5) representing the object 40 in the virtual space 46. The mass point 45 is a point that has mass and no size. The virtual space 46 is a two-dimensional space in which the value in the z-axis direction is a predetermined fixed value in a coordinate system consisting of three axes of the x-axis, y-axis, and z-axis in the real space. The processor 23 may perform mapping conversion of the coordinates of the tracked mass point 45 in the virtual space 46 into coordinates in the moving image.

[0025] Moreover, the processor 23 detects a detection target from a moving image and tracks it using a Kalman filter. Here, the processor 23 can detect a plurality of detection targets from a moving image and tracks each of the plurality of detection targets using a Kalman filter. When detecting a plurality of detection targets, if their images overlap in the moving image, tracking may be erroneous or accuracy may decrease in the conventional technology. In this embodiment, the processor 23 can avoid such a problem by associating one or more Kalman filters with each of the plurality of detection targets. Furthermore, the processor 23 manages the observation value, the Kalman filter, and the unique identification information of the tracked object (hereinafter, "tracked object ID") in each layer. The processor 23 determines whether the tracked objects are the same object (same detection target) or not, and executes a process of associating the observation value, the Kalman filter, and the tracked object ID. This can further improve the accuracy of tracking the plurality of detection targets.

[0026] Furthermore, the processor 23 initializes or selects a low-probability Kalman filter among a plurality of Kalman filters associated with the same detection target or observation value as a candidate for exclusion from the association. For example, when many Kalman filters are associated with the same detection target or observation value, the processor 23 can prevent an increase in the computation load by excluding some of the Kalman filters. Details of the processing performed by the processor 23 will be described later. The processor 23 may include a plurality of processors, similar to the processor 13 of the imaging device 10. The processor 23 may be configured by combining a plurality of types of devices, similar to the processor 13.

[0027] The output interface 24 is configured to output an output signal from the object tracking device 20. The output interface 24 may be referred to as an output unit. The output interface 24 may output a detection result of the detection target, such as the coordinates of the mass point 45.

[0028] The output interface 24 may include a physical connector and a wireless communication device. The output interface 24 may be connected to a network of the vehicle 100, such as a CAN. The output interface 24 may be connected to the display 30, a control device, an alarm device, and the like of the vehicle 100 via a communication network such as a CAN. The information output from the output interface 24 may be used as appropriate by each of the display 30, the control device, and the alarm device.

[0029] The display 30 may display a moving image output from the object tracking device 20. When the display 30 receives coordinates of a mass point 45 representing the position of an image of an object from the object tracking device 20, the display 30 may have a function of generating an image element (e.g., a warning to be displayed together with an approaching object) according to the coordinates and superimposing the image on the moving image. The display 30 may employ various types of devices. For example, the display 30 may employ a liquid crystal display (LCD), an organic electro-luminescence (EL) display, an inorganic EL display, a plasma display panel (PDP), a field emission display (FED), an electrophoretic display, a twist ball display, or the like.

[0030] Next, an object tracking method executed by the object tracking device 20 will be described with reference to the flowchart of FIG. 3. The object tracking device 20 may be configured to read and implement a program recorded in a non-transitory computer-readable medium to execute the process performed by the processor 23 described below. The non-transitory computer-readable medium includes, but is not limited to, a magnetic storage medium, an optical storage medium, a magneto-optical storage medium, and a semiconductor storage medium. The magnetic storage medium includes a magnetic disk, a hard disk, and a magnetic tape. The optical storage medium includes an optical disk such as a CD, a DVD, and a BD. The semiconductor storage medium includes a read only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory.

[0031] The flowchart in FIG. 3 shows the process executed by the processor 23 by acquiring successive frames of a video. The processor 23 of the object tracking device 20 tracks the position of an object image 42 (see FIG. 4) every time a frame of a video is acquired according to the flowchart in FIG. 3. As shown in FIG. 2, there may be a plurality of objects 40 to be detected, and may include a pedestrian 40A, an automobile 40B, and a bicycle 40C. Furthermore, the object 40 is not limited to moving objects and people, and may include various targets such as obstacles on a road. In the following description of the object tracking method, one of the plurality of objects 40 (specifically, a pedestrian 40A) included in the video of the imaging device 10 installed behind the vehicle 100 will be used for the description. The other objects 40 (for example, an automobile 40B and a bicycle 40C) are also tracked by the same process.

[0032] The processor 23 acquires each frame of a moving image from the imaging device 10 via the input interface 21 (step S101). An example of one frame of a moving image is shown in FIG. 4. In the example of FIG. 4, an image of an object 40 (object image 42) behind the vehicle 100 is displayed in a two-dimensional image space 41 formed of a uv coordinate system. The u coordinate is the horizontal coordinate of the image. The v coordinate is the vertical coordinate of the image. In FIG. 4, the origin of the uv coordinate is the upper left corner point of the image space 41. The positive direction of the u coordinate is from left to right. The positive direction of the v coordinate is from top to bottom.

[0033] The processor 23 recognizes the image 42 of the object from each frame of the video by image recognition (step S102). Methods for recognizing the image 42 of the object include various known methods. For example, methods for recognizing the image 42 of the object include a method that uses shape recognition of objects such as cars and people, a template matching method, and a method that calculates features from an image and uses them for matching. A function approximator capable of learning the relationship between input and output can be used to calculate the features. A neural network, for example, can be used as the function approximator that can learn the relationship between input and output.

[0034] The processor 23 maps and transforms the coordinates (u, v) of the image 42 of the object in the image space 41 into coordinates (x', y') of the object in the virtual space 46 (see FIG. 6) (step S103). In general, the coordinates (u, v) of the image space 41, which are two-dimensional coordinates, cannot be transformed into coordinates (x, y, z) of the real space. However, by specifying the height in the real space and fixing the z coordinate to a predetermined value, the coordinates (u, v) of the image space 41 can be transformed into coordinates (x, y, z) of the real space. 0 )(z 0 It is possible to map the coordinates (x', y') of a two-dimensional virtual space 46 corresponding to the input information (x, y are fixed values). In this embodiment, the virtual space 46 is two-dimensional, but it may be three-dimensional depending on the input information (type of sensor).

[0035] 4, a representative point 43 located at the center of the bottom of the object image 42 is identified. For example, the representative point 43 can be the lowest position of the v coordinate and the center position of the range of the u coordinate of the area occupied by the object image 42 in the image space 41. This representative point 43 is assumed to be a position of the object 40 corresponding to the object image 42 that is in contact with the road surface or ground.

[0036] FIG. 5 shows the relationship between an object 40 located in a three-dimensional real space and an image 42 of the object in a two-dimensional image space 41. When the internal parameters of the imaging device 10 are known, the direction from the center of the imaging optical system 11 of the imaging device 10 to the corresponding coordinates (x, y, z) in the real space can be calculated based on the coordinates (u, v) in the image space 41. The internal parameters of the imaging device 10 include information such as the focal length and distortion of the imaging optical system 11 and the pixel size of the imaging element 12. In the real space, a point where a straight line directed toward a direction corresponding to a representative point 43 in the image space 41 intersects with a reference plane 44 of z=0 is defined as a mass point 45 of the object 40. The reference plane 44 corresponds to the road surface or ground on which the vehicle 100 is located. The mass point 45 has three-dimensional coordinates (x, y, 0). Therefore, when the two-dimensional space of z=0 is defined as a virtual space 46, the coordinates of the mass point 45 can be expressed as (x', y'). The coordinates (x', y') of mass point 45 in virtual space 46 correspond to the coordinates (x, y) of a specific point of object 40 on the xy plane (z=0) when object 40 is viewed from the direction along the z-axis in real space. The specific point is a point that corresponds to mass point 45.

[0037] As shown in FIG. 6, the processor 23 calculates the position (x', y') and velocity (v x´ ,v y´ ) is tracked (step S104). x´ ,v y´ By having information on the position (x', y') of mass point 45 in the successive frames, processor 23 can predict the range of the position (x', y') of mass point 45 in the successive frames. Processor 23 can recognize mass point 45 located in the predicted range in the next frame as mass point 45 corresponding to image 42 of the tracked object. Each time a new frame is input, processor 23 sequentially calculates the position (x', y') and velocity (v x´ ,v y´ ) to update the

[0038] For tracking the mass point 45, for example, estimation using a Kalman filter based on a state space model can be adopted. By performing prediction / estimation using a Kalman filter, robustness against failure to detect and false detection of the object 40 to be detected is improved. In general, it is difficult to describe the image 42 of the object in the image space 41 with an appropriate model that describes the motion. Therefore, it has been difficult to easily estimate the position of the image 42 of the object in the image space 41 with high accuracy. In the object tracking device 20 of the present disclosure, the image 42 of the object is converted into a mass point 45 in the real space, which makes it possible to apply a model that describes the motion in the real space, thereby improving the accuracy of tracking the image 42 of the object. In addition, by treating the object 40 as a mass point 45 that does not have a size, simple and easy tracking is possible.

[0039] Each time the processor 23 estimates a new position of the mass point 45, the processor 23 may map and transform the coordinates of the mass point 45 in the virtual space 46 into coordinates (u, v) in the image space 41 to indicate the estimated position (step S105). The mass point 45 located at coordinates (x', y') in the virtual space 46 can be mapped and transformed into the image space 41 as a point located at coordinates (x, y, 0) in the real space. The coordinates (x, y, 0) in the real space can be mapped into coordinates (u, v) in the image space 41 of the imaging device 10 by a known method. The processor 23 can convert between the coordinates (u, v) in the image space 41, the coordinates (x', y') in the virtual space 46, and the coordinates (x, y, 0) in the real space.

[0040] (Data Association) FIG. 7 is a diagram for explaining data association. Data association is a process of associating a Kalman filter with an observation value. In data association, a plurality of Kalman filters can be associated with a plurality of observation values. Here, the observation value is the position of the detection target. The processor 23 assigns identifiers to the plurality of observation values ​​and the plurality of Kalman filters to distinguish them. In this embodiment, the processor 23 uses, for example, serial numbers to name the plurality of observation values ​​as observation value (1), observation value (2), observation value (3), etc. The processor 23 also uses, for example, symbols and serial numbers to name the plurality of Kalman filters as KF(1), KF(2), KF(3), etc.

[0041] In this embodiment, the processor 23 performs data association between M observations and N Kalman filters. M is an integer equal to or greater than 2. N is an integer equal to or greater than M. In the example of FIG. 7, the processor 23 performs data association between three observations and five Kalman filters. Observation (1) is the position of a pedestrian 40A detected in frame (k) of the video. Observation (2) is the position of a car 40B detected in frame (k) of the video. Observation (3) is the position of a bicycle 40C detected in frame (k) of the video. Frame (k-1) is the frame immediately preceding frame (k) in the video. Frame (k-2) is the frame immediately preceding frame (k) in the video. The current frame is assumed to be frame (k).

[0042] Here, KF(2) was used to track pedestrian 40A up until frame (k-1), but is initialized because it has satisfied the initialization condition described below, and is no longer used to track the position of the detection target. KF(5) is a new Kalman filter that was prepared when a new bicycle 40C was recognized in frame (k-2). KF(5) has started tracking the detection target because the newly recognized bicycle 40C was also recognized in the current frame (k). The other Kalman filters have continued tracking the detection targets since frame (k-2).

[0043] In the example of FIG. 7, the processor 23 associates KF(1) with the observation value (1). The processor 23 associates KF(3) and KF(4) with the observation value (2). The processor 23 also associates KF(5) with the observation value (3). As in the example of the observation value (2), the processor 23 allows overlapping of detection results in the process of tracking multiple detection targets. That is, the processor 23 uses KF(3) and KF(4) to predict the range of the observation value (2), i.e., the position of the automobile 40B. In this way, by allowing overlapping in the data association, local optimization can be performed. For example, a method (e.g., the Hungarian method) in which multiple observation values ​​are associated one-to-one with multiple Kalman filters without allowing overlapping may cause a chain of misassociations due to global optimization. In this embodiment, since overlapping is allowed, problems such as chaining of misassociations do not occur. Furthermore, in the tracking process, one or more Kalman filters are associated with one observed value, making it difficult for tracking failures to occur for any observed value, thereby improving robustness.

[0044] (Tracked object ID management) Here, as described above, a plurality of Kalman filters may be associated with one observation value, but a plurality of observation values ​​may also be associated with one object to be detected. For example, when the detection object is an automobile 40B, and the automobile 40B disappears from the video due to a lane change or the like and then reappears in the video, a new observation value may be associated with the automobile 40B as a different object. In order to accurately track objects, it is preferable that the object tracking device 20 identifies each tracked object and grasps the correspondence with the observation value. In this embodiment, the processor 23 executes tracked object ID management using a hierarchical structure as described below, and groups a plurality of Kalman filters to determine whether they correspond to the same object.

[0045] FIG. 8 is a diagram showing a hierarchical structure of tracked object ID management (ID management) in this embodiment. Tracked object ID management is a process of associating a Kalman filter with a detection target. As shown in FIG. 8, the processor 23 manages an observation value, a Kalman filter, and a tracked object ID in each layer. The processor 23 also enables accurate tracking of an object by associating an observation value, a Kalman filter, and a tracked object ID. Here, the tracked object ID is unique identification information of a tracked object as described above. If the tracked object ID associated with multiple observation values ​​or multiple Kalman filters is the same, these observation values ​​or Kalman filters are related to tracking of the same object.

[0046] When a video frame is acquired, the processor 23 executes grouping of a plurality of Kalman filters. Then, the processor 23 updates the correspondence between the observation value, the Kalman filter, and the tracked object ID. In the example of FIG. 8, the processor 23 groups KF(1), KF(2), and KF(3), assigns an identifier "tracked object ID(1)" to the object to be tracked using these Kalman filters, and performs tracking control of the object. The processor 23 also groups KF(4) and KF(5), assigns an identifier "tracked object ID(2)" to the object to be tracked using these Kalman filters, and performs tracking control of the object. The processor 23 links the Kalman filters corresponding to the objects determined to be the same, and controls tracking in a hierarchical structure that also links the detection results of the detection targets corresponding to these Kalman filters, thereby enabling error-free and highly accurate tracking. The processor 23 can obtain a detection result with a high degree of certainty by, for example, comparing or selecting detection results using the linked Kalman filters.

[0047] (Overlapping Kalman Filter Management) As described above, multiple Kalman filters can be associated with one observation value, and multiple Kalman filters can be associated with one detection target (a detection target having one tracked object ID). By associating multiple Kalman filters, tracking failures are less likely to occur, and robustness can be improved. However, associating many Kalman filters increases the computational load and can cause delays in the control of the object tracking device 20 by the processor 23. The processor 23 performs overlapping Kalman filter management to exclude some overlapping Kalman filters (multiple Kalman filters associated with the same detection target or observation value) from the association, as described below.

[0048] In this embodiment, the processor 23 executes a first process and a second process as overlapping Kalman filter management. In the first process, the processor 23 selects an overlapping Kalman filter with a low "likelihood" as a "candidate Kalman filter to be excluded" that can be excluded from the association. The first process can be likened to pruning a Kalman filter with low importance, and can be called a pruning process. In the second process, the processor 23 excludes a "candidate Kalman filter to be excluded" that satisfies an initialization condition from the association and initializes it. In the second process, the Kalman filter is initialized to an empty state, and can be called a vacant process.

[0049] Here, the "likelihood" of the Kalman filter in the first process refers to the degree of accuracy of prediction / estimation of the position of the associated detection target or observed value, in other words, the degree of certainty. The processor 23 can determine the likelihood of the Kalman filter, for example, by the size of the error ellipse. The error ellipse indicates an estimation range based on a probability density distribution of the position, and indicates that the position is located inside the ellipse with a predetermined probability (for example, 99%). The error ellipse is calculated using the standard deviation in the x' direction and the standard deviation in the y' direction of the two-dimensional virtual space 46 (see FIG. 6). Of the overlapping Kalman filters, the Kalman filter with the largest error ellipse is considered to have low likelihood and can be selected as a "candidate Kalman filter to be excluded."

[0050] Moreover, the initialization condition in the second process is that the number of times selected as the candidate Kalman filter for exclusion reaches a first value. The first value may be selected arbitrarily, and is, for example, "5". For example, when a certain Kalman filter is selected as the candidate Kalman filter for exclusion five times by the first process, the second process is executed, and the Kalman filter that became the fifth candidate Kalman filter for exclusion is excluded from the correspondence with the detection target or the observation value, and is initialized. Here, the number of times selected as the candidate Kalman filter for exclusion may be consecutive or may be cumulative. For example, when the number of consecutive times is used, if a certain Kalman filter is not selected as the candidate Kalman filter for exclusion in the first process, the number returns to "0". The number of times selected as the candidate Kalman filter for exclusion may be counted for each Kalman filter by a counter provided in the processor 23.

[0051] The processor 23 may execute the first process and the second process in the above data association. FIG. 9 is a diagram showing the error ellipses of the overlapping Kalman filters associated with the same observation point in the data association. A situation in which multiple Kalman filters are associated with the same observation point (one observation value) may include, for example, a case in which one object is recognized as two objects due to the influence of light reflection, and a new Kalman filter is associated with one of the objects. As described above, the control of tracking of the detection target may be executed in parallel using the associated multiple Kalman filters, but the amount of calculation processing may increase. Therefore, it is preferable to execute overlapping Kalman filter management in the data association.

[0052] In the example of FIG. 9, three Kalman filters, KF(p), KF(q) and KF(r), are associated with the same observation point. The processor 23 calculates an error ellipse for each of the three Kalman filters. The processor 23 executes a first process to select a Kalman filter having a relatively large error ellipse as a candidate Kalman filter to be excluded. The processor 23 may select multiple candidate Kalman filters to be excluded, but in the example of FIG. 9, the processor 23 selects KF(q) having the largest error ellipse as a candidate Kalman filter to be excluded. Thereafter, the processor 23 executes a second process. In the example of FIG. 9, when the number of times that KF(q) has been selected as a candidate Kalman filter to be excluded reaches a first value (e.g., 5 times), that is, when the initialization condition is satisfied, the processor 23 removes the association of KF(q) and initializes KF(q).

[0053] Here, the processor 23 may execute the first process and the second process when the number of multiple Kalman filters associated with the same observation value exceeds a second value. The second value may be selected arbitrarily, but is "2" in the example of FIG. 9. The second value is an upper limit value of the number of overlapping Kalman filters associated with the same observation point. The second value is preferably a small value from the viewpoint of reducing the burden of calculation processing, but is set to 2 or more to ensure robustness in the processing of tracking an object. In the example of FIG. 9, since the number of overlapping Kalman filters exceeds 2, the processor 23 executes the first process and the second process to initialize KF(q). By excluding KF(q) with a relatively low importance from the association, an increase in the calculation processing of the processor 23 is prevented. In addition, KF(p) and KF(r) continue to be associated with the same observation point, and robustness is also ensured in tracking the position of the detection target corresponding to this observation point.

[0054] In the example of FIG. 9, one KF(q) is selected as the exclusion candidate Kalman filter, but multiple exclusion candidate Kalman filters may be selected in the first process. Similarly, multiple exclusion candidate Kalman filters may be initialized in the second process. The number of exclusion candidate Kalman filters may be determined based on the number of overlapping Kalman filters and the above-mentioned second value (the upper limit of the number of overlapping Kalman filters associated with the same observation point). For example, when the second value is "2" and the number of overlapping Kalman filters is "5", the processor 23 may select three exclusion candidate Kalman filters, which are the differences between these, in the first process. In other words, the processor 23 may select three Kalman filters with relatively large error ellipses as candidates to be excluded from association with the same observation point in the first process.

[0055] Moreover, the processor 23 may execute a first process and a second process in the above-mentioned tracking object ID management. Fig. 10 is a diagram showing error ellipses of the respective overlapping Kalman filters associated with the same detection target in the tracking object ID management. As described above, the control of tracking of the detection target can be executed in parallel using a plurality of Kalman filters associated with the same detection target (the same object having one tracking object ID), but the amount of calculation processing may increase. Therefore, it is preferable to execute overlapping Kalman filter management in the tracking object ID management.

[0056] Here, in the tracking object ID management, the same detection target and the Kalman filter are associated with each other by clustering such as DBSCAN (density-based spatial clustering of applications with noise). As shown in FIG. 10, when the centers of the error ellipses of a plurality of Kalman filters are included in a predetermined range, the processor 23 determines that the Kalman filters belong to one group. In the example of FIG. 10, the predetermined range is indicated by a circle. Here, the predetermined range may change according to the size of the tracking object. For example, if the tracking object is an automobile 40B, the predetermined range may be set larger than when the tracking object is a pedestrian 40A. In addition, the predetermined range may be constant regardless of the type of tracking object. The clustering method is not limited to DBSCAN. For example, clustering may be performed by other methods such as the k-means method.

[0057] In the example of Fig. 10, three Kalman filters, KF(p), KF(q) and KF(r), are associated with the same detection target. As in the example of Fig. 9, the processor 23 executes a first process to select a Kalman filter having a relatively large error ellipse as a candidate Kalman filter to be excluded. Thereafter, the processor 23 executes a second process. In the example of Fig. 10, when the number of times that KF(q) has been selected as a candidate Kalman filter to be excluded reaches a first value (e.g., 5 times), that is, when the initialization condition is satisfied, the processor 23 removes the association of KF(q) and initializes KF(q).

[0058] Here, the processor 23 may execute the first process and the second process when the number of multiple Kalman filters associated with the same detection target exceeds the second value, as in the example of FIG. 9. The second value is the "upper limit value of the number of overlapping Kalman filters associated with the same observation point" as described above, and is "2" as an example. In the example of FIG. 10, since the number of overlapping Kalman filters exceeds 2, the processor 23 executes the first process and the second process to initialize KF(q). By excluding KF(q) with a relatively low importance from the association, an increase in the calculation process of the processor 23 is prevented. In addition, KF(p) and KF(r) continue to be associated with the same detection target, and robustness is ensured in tracking the position of this detection target.

[0059] Here, as another example, the processor 23 may execute the first process and the second process when the number of multiple Kalman filters associated with the same detection target exceeds a third value. The third value is an "upper limit value of the number of overlapping Kalman filters associated with the same detection target" and is set regardless of the second value. For example, the processor 23 may execute the first process and the second process in the data association when the number of overlapping Kalman filters exceeds a second value ("2" as an example), and may execute the first process and the second process in the tracked object ID management when the number of overlapping Kalman filters exceeds a third value ("4" as an example).

[0060] The processor 23 may execute the overlapping Kalman filter management by selecting or combining the above execution timings and execution conditions. The processor 23 may execute the first process and the second process, for example, in the data association and the tracked object ID management. The processor 23 may execute the first process and the second process, for example, when the number of overlapping Kalman filters exceeds a second value, only in the data association. The processor 23 may also execute the first process and the second process, for example, when the number of overlapping Kalman filters exceeds a third value, only in the tracked object ID management.

[0061] As described above, the object tracking device 20 according to this embodiment allows overlapping of detection results in the process of tracking multiple detection targets by the above configuration. Therefore, the object tracking device 20 can track multiple objects with high accuracy without causing a chain of misassociations. Furthermore, the object tracking device 20 according to this embodiment also executes overlapping Kalman filter management in the process of tracking objects. Therefore, the object tracking device 20 can track objects with high accuracy without increasing the computation load.

[0062] Although the embodiments according to the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions included in each component or each step can be rearranged so as not to be logically contradictory, and multiple components or steps can be combined into one or divided. Although the embodiments according to the present disclosure have been described mainly with respect to the device, the embodiments according to the present disclosure can also be realized as a method including steps executed by each component of the device. The embodiments according to the present disclosure can also be realized as a method, a program, or a storage medium on which a program is recorded, executed by a processor provided in the device. It should be understood that these are also included in the scope of the present disclosure. For example, the overlapping Kalman filter management can also be realized as a method by including each step of the processing executed by the processor 23 in the object tracking method of FIG. 3.

[0063] In the above embodiment, the object tracking system 1 includes the imaging device 10, the object tracking device 20, and the display 30, but at least two of these may be integrated. For example, the function of the object tracking device 20 may be mounted on the imaging device 10. In this case, the imaging device 10 may include the above-mentioned storage unit 22 and output interface 24 in addition to the imaging optical system 11, the imaging element 12, and the processor 13. Furthermore, the processor 13 may execute the processing performed by the processor 23 in the above embodiment on the moving image output by the imaging device 10. With such a configuration, the imaging device 10 that performs object tracking may be realized.

[0064] In the present disclosure, the term "mobile body" includes vehicles, ships, and aircraft. In the present disclosure, the term "vehicle" includes, but is not limited to, automobiles and industrial vehicles, and may include railroad vehicles, lifestyle vehicles, and fixed-wing aircraft that run on runways. Automobiles include, but are not limited to, passenger cars, trucks, buses, motorcycles, trolley buses, and the like, and may include other vehicles that run on roads. Industrial vehicles include industrial vehicles for agriculture and construction. Industrial vehicles include, but are not limited to, forklifts and golf carts. Industrial vehicles for agriculture include, but are not limited to, tractors, cultivators, transplanters, binders, combines, and lawnmowers. Industrial vehicles for construction include, but are not limited to, bulldozers, scrapers, excavators, cranes, dump trucks, and road rollers. Vehicles include those that run by human power. Here, the classification of vehicles is not limited to the above. For example, automobiles may include industrial vehicles that can run on roads, and the same vehicle may be included in multiple classifications. In the present disclosure, the vessel includes a marine jet, a boat, and a tanker. In the present disclosure, the aircraft includes a fixed-wing aircraft and a rotorcraft. [Explanation of symbols]

[0065] 1. Object Tracking System 10. Imaging device 11 Imaging optical system 12 Image sensor 13 Processors 20 Object Tracking Device 21 Input Interface 22 Memory section 23 Processors 24 Output Interfaces 30 Display 40 Object 40A Pedestrian 40B car 40C Bicycle 41 Image Space 42 Image of an object 43 Representative points 44 Reference plane 45 mass points 46 Virtual Space 100 vehicles

Claims

1. an input interface for acquiring sensor data; a processor that detects a detection target from the sensor data and tracks the detection target using a Kalman filter in which unique identification information of the detection target and observed values ​​that are the position of the detection target are associated with each other; an output interface for outputting a detection result of the detection object; The processor, a first process of selecting, from among a plurality of Kalman filters associated with the same detection target or observation value, a filter having a low degree of certainty, i.e., a degree of certainty of the associated detection target or observation value, as a candidate Kalman filter to be excluded from the association; and a second process of initializing the excluded candidate Kalman filter that satisfies an initialization condition from the correspondence.

2. The object tracking device according to claim 1 , wherein the processor determines the likelihood of the Kalman filter based on the size of an error ellipse.

3. The object tracking device according to claim 1 , wherein the initialization condition is that the number of times the Kalman filter is selected as the excluded candidate Kalman filter reaches a first value.

4. The object tracking device according to any one of claims 1 to 3, wherein the processor executes the first process and the second process when the number of the multiple Kalman filters associated with the same detection target or observation value exceeds a second value.

5. The object tracking device according to claim 1 , wherein the processor executes the first process and the second process in a process of associating the Kalman filter with the observation value.

6. The object tracking device according to claim 1 , wherein the processor executes the first process and the second process in a process of associating the Kalman filter with the detection target.

7. Acquiring sensor data; detecting a detection target from the sensor data, and tracking the detection target using a Kalman filter in which unique identification information of the detection target and an observation value that is a position of the detection target are associated with each other; and outputting a detection result of the detection target. The tracking of the detection target includes: a first process of selecting, from among a plurality of Kalman filters associated with the same detection target or observation value, a filter having a low degree of certainty, i.e., a degree of certainty of the associated detection target or observation value, as a candidate Kalman filter to be excluded from the association; a second process of initializing the excluded candidate Kalman filter that satisfies an initialization condition from the correspondence.

Images