Object Tracking Device

Abstract

To provide an object tracking device that can accurately and stably track an object.SOLUTION: An object tracking device (20) comprises: an input interface (21) that acquires sensor data; a processor (23) that detects an object to be detected from the sensor data and tracks the object to be detected by using Kalman filters associated with the object to be detected and an observed value; and an output interface (24) that outputs a result of detection of the object to be detected. The processor (23) puts restrictions on a variation range of indices of the Kalman filters affecting the tracking of the object to be detected.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an object tracking device. [Background technology]

[0002] There are known techniques for detecting surrounding objects, tracking the detected objects, and predicting their movements. For example, Patent Document 1 discloses a device that processes video signals output from an on-board camera that captures video around a vehicle, detects the presence or absence of approaching vehicles and pedestrians, and displays the approaching vehicles and pedestrians by adding square frame marks. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-321494 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, vehicles detected as separate in the image may actually be the same vehicle, and vice versa, vehicles detected as a single vehicle may actually be separate vehicles. Therefore, there is a need for a technology that can continue tracking with higher accuracy and without confusion.

[0005] In view of the above, an object of the present disclosure is to provide an object tracking device that can stably track an object with high accuracy. [Means for solving the problem]

[0006] An object tracking device according to an embodiment includes: an input interface for acquiring sensor data; a processor that detects a detection target from the sensor data and tracks the detection target using a Kalman filter in which the detection target and an observation value are associated with each other; an output interface for outputting a detection result of the detection object; The processor sets a limit on a range of variation of an index of the Kalman filter that affects tracking of the detected object. Effect of the Invention

[0007] According to an embodiment of the present disclosure, it is possible to provide an object tracking device that can stably track an object with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of an object tracking system including an object tracking device according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing a vehicle equipped with the object tracking system of FIG. 1 and a detection target. [Diagram 3] FIG. 3 is a flowchart showing an example of a process for tracking an image of an object on a moving image. [Figure 4] FIG. 4 is a diagram showing an example of an image of an object on a moving image. [Diagram 5] FIG. 5 is a diagram for explaining the relationship between an object in real space, an image of the object in a moving image, and a mass point in virtual space. [Figure 6] FIG. 6 is a diagram showing an example of movement of a mass point in a virtual space. [Figure 7] FIG. 7 is a diagram for explaining data association. [Figure 8] FIG. 8 is a diagram illustrating a hierarchical structure of tracking object ID management. [Figure 9] FIG. 9 is a diagram for explaining the restriction on the Mahalanobis distance. [Figure 10] FIG. 10 is a diagram for explaining the restriction of the grouping area. [Figure 11] FIG. 11 is a diagram for explaining the restriction of the grouping area. [Figure 12] FIG. 12 is a diagram for explaining limitations on the size of the error ellipse according to the output accuracy. [Figure 13] FIG. 13 is a diagram for explaining the limitation of the size of the error ellipse according to the saturation of the detection result. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The drawings used in the following description are schematic. The dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0010] Fig. 1 is a block diagram showing a schematic configuration of an object tracking system 1. An object tracking device 20 according to an embodiment of the present disclosure is included in the object tracking system 1. In this embodiment, the object tracking system 1 includes an imaging device 10, the object tracking device 20, and a display 30. In this embodiment, the object tracking system 1 is mounted on a vehicle 100, which is an example of a moving body, as illustrated in Fig. 2.

[0011] The object tracking device 20 according to this embodiment acquires a moving image from the imaging device 10 as sensor data. That is, in this embodiment, the sensor used to detect the detection target is the imaging element 12 that captures visible light provided in the imaging device 10. However, the object tracking system 1 is not limited to the configuration shown in FIG. 1. The object tracking system 1 can be provided with a device different from the imaging device 10 as long as it detects the detection target. As another example, the object tracking system 1 may be configured to include a measuring device that measures the distance to the detection target from the reflected wave of the irradiated laser light instead of the imaging device 10. As another example, the object tracking system 1 may be configured to include a detection device having a millimeter wave sensor instead of the imaging device 10. Also, as another example, the object tracking system 1 may be configured to include an imaging device 10 that includes an imaging element 12 that captures light other than the visible light region.

[0012] In this embodiment, the object tracking system 1 is mounted on a moving body, and detects objects 40 (see FIG. 2) around the moving moving body. However, the object tracking system 1 is not limited to being mounted on a moving body. As another example, the object tracking system 1 may be used in a facility such as a factory, and detect employees, transport robots, and products. As another example, the object tracking system 1 may be used in an elderly welfare facility, and detect elderly people and staff in the room. The object tracking system 1 may not only track objects for the safety of driving or activities, but also track objects for the efficiency of work, quality control, or productivity improvement in agricultural and industrial sites. Here, in the present disclosure, objects that are detected by the object tracking device 20 include not only objects such as moving bodies, but also people.

[0013] As shown in FIG. 2, in this embodiment, the x-axis direction of the coordinates of the real space is the width direction of the vehicle 100 on which the imaging device 10 is installed. The positive y-axis direction is the direction in which the vehicle 100 moves backward. The x-axis and y-axis directions are parallel to the road surface on which the vehicle 100 is located. The z-axis direction is perpendicular to the road surface. The z-axis direction can be called the vertical direction. The x-axis, y-axis, and z-axis directions are mutually orthogonal. The way in which the x-axis, y-axis, and z-axis directions are taken is not limited to this. The x-axis, y-axis, and z-axis directions can be interchanged with each other.

[0014] The imaging device 10 includes an imaging optical system 11, an imaging element 12, and a processor 13.

[0015] The imaging device 10 may be installed at various positions on the vehicle 100. The imaging device 10 includes, but is not limited to, a front camera, a left side camera, a right side camera, and a rear camera. The front camera, the left side camera, the right side camera, and the rear camera are installed on the vehicle 100 so as to be able to capture images of the surrounding areas in front of, on the left side of, on the right side of, and behind the vehicle 100, respectively. In an embodiment described below as an example, as shown in FIG. 2, the imaging device 10 is attached to the vehicle 100 with its optical axis direction facing downward from the horizontal direction so as to be able to capture images of the rear of the vehicle 100.

[0016] The imaging optical system 11 may include one or more lenses. The imaging element 12 may include a charge-coupled device image sensor (CCD image sensor) or a complementary MOS image sensor (CMOS image sensor).

[0017] The imaging element 12 converts an image of an object (subject image) formed on the imaging surface of the imaging element 12 by the imaging optical system 11 into an electrical signal. The imaging element 12 can capture moving images at a predetermined frame rate. A frame is each still image that constitutes a moving image. The number of images that can be captured per second is called the frame rate. The frame rate may be, for example, 60 fps (frames per second) or 30 fps.

[0018] The processor 13 controls the entire imaging device 10 and performs various image processing on the moving images output from the imaging element 12. The image processing performed by the processor 13 may include any processing such as distortion correction, brightness adjustment, contrast adjustment, gamma correction, etc.

[0019] The processor 13 may be comprised of one or more processors. The processor 13 includes one or more circuits or units configured to perform one or more data computing procedures or processes, for example, by executing instructions stored in associated memory. The processor 13 includes one or more processors, microprocessors, microcontrollers, application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), any combination of these devices or configurations, or other known devices or configurations.

[0020] The object tracking device 20 includes an input interface 21, a memory unit 22, a processor 23, and an output interface 24.

[0021] The input interface 21 is configured to be able to communicate with the imaging device 10 via a wired or wireless communication means. The input interface 21 acquires moving images from the imaging device 10 as sensor data. The input interface 21 may correspond to a transmission method of an image signal transmitted by the imaging device 10. The input interface 21 can be referred to as an input unit or an acquisition unit. The imaging device 10 and the input interface 21 may be connected via an in-vehicle communication network such as a control area network (CAN).

[0022] The storage unit 22 is a storage device that stores data and programs necessary for processing performed by the processor 23. For example, the storage unit 22 temporarily stores moving images acquired from the imaging device 10. For example, the storage unit 22 stores data generated by processing performed by the processor 23. The storage unit 22 may be configured using, for example, one or more of a semiconductor memory, a magnetic memory, an optical memory, and the like. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk and a magnetic tape. The optical memory may include, for example, a compact disc (CD), a digital versatile disc (DVD), and a Blu-ray (registered trademark) disc (BD).

[0023] The processor 23 controls the entire object tracking device 20. The processor 23 recognizes an image of an object included in a moving image acquired via the input interface 21. The processor 23 performs mapping conversion of the coordinates of the recognized image of the object into the coordinates of the object 40 in a virtual space 46 (see FIG. 6), and tracks the position and velocity of a mass point 45 (see FIG. 5) representing the object 40 in the virtual space 46. The mass point 45 is a point that has mass and no size. The virtual space 46 is a two-dimensional space in which the value in the z-axis direction is a predetermined fixed value in a coordinate system consisting of three axes of the x-axis, y-axis, and z-axis in the real space. The processor 23 may perform mapping conversion of the coordinates of the tracked mass point 45 in the virtual space 46 into coordinates in the moving image.

[0024] Moreover, the processor 23 detects a detection target from a moving image and tracks it using a Kalman filter. Here, the processor 23 can detect a plurality of detection targets from a moving image and tracks each of the plurality of detection targets using a Kalman filter. When detecting a plurality of detection targets, if their images overlap in the moving image, tracking may be erroneous or accuracy may decrease in the conventional technology. In this embodiment, the processor 23 can avoid such a problem by associating one or more Kalman filters with each of the plurality of detection targets. Furthermore, the processor 23 manages the observation value, the Kalman filter, and the unique identification information of the tracked object (hereinafter, "tracked object ID") in each layer. The processor 23 determines whether the tracked objects are the same object (same detection target) or not, and executes a process of associating the observation value, the Kalman filter, and the tracked object ID. This can further improve the accuracy of tracking the plurality of detection targets.

[0025] Moreover, the processor 23 sets a limit on the range of variation of the index of the Kalman filter that affects the tracking of the detection target. In this embodiment, the processor 23 sets at least one of an upper limit and a lower limit for an index including at least one of the Mahalanobis distance used for associating the Kalman filter with the observation value, the radius of the grouping area used for associating the Kalman filter with the detection target, and the size of the error ellipse of the Kalman filter. By appropriately limiting the range of variation of the index of the Kalman filter, the association between the detection target and the observation value is made less likely to be interrupted, enabling stable tracking. Details of the processing performed by the processor 23 will be described later. The processor 23 may include multiple processors, similar to the processor 13 of the imaging device 10. Also, the processor 23 may be configured by combining multiple types of devices, similar to the processor 13.

[0026] The output interface 24 is configured to output an output signal from the object tracking device 20. The output interface 24 may be referred to as an output unit. The output interface 24 may output a detection result of the detection target, such as the coordinates of the mass point 45.

[0027] The output interface 24 may include a physical connector and a wireless communication device. The output interface 24 may be connected to a network of the vehicle 100, such as a CAN. The output interface 24 may be connected to the display 30, a control device, an alarm device, and the like of the vehicle 100 via a communication network such as a CAN. The information output from the output interface 24 may be used as appropriate by each of the display 30, the control device, and the alarm device.

[0028] The display 30 may display a moving image output from the object tracking device 20. When the display 30 receives coordinates of a mass point 45 representing the position of an image of an object from the object tracking device 20, the display 30 may have a function of generating an image element (e.g., a warning to be displayed together with an approaching object) according to the coordinates and superimposing the image on the moving image. The display 30 may employ various types of devices. For example, the display 30 may employ a liquid crystal display (LCD), an organic electro-luminescence (EL) display, an inorganic EL display, a plasma display panel (PDP), a field emission display (FED), an electrophoretic display, a twist ball display, or the like.

[0029] Next, an object tracking method executed by the object tracking device 20 will be described with reference to the flowchart of FIG. 3. The object tracking device 20 may be configured to read and implement a program recorded in a non-transitory computer-readable medium to execute the process performed by the processor 23 described below. The non-transitory computer-readable medium includes, but is not limited to, a magnetic storage medium, an optical storage medium, a magneto-optical storage medium, and a semiconductor storage medium. The magnetic storage medium includes a magnetic disk, a hard disk, and a magnetic tape. The optical storage medium includes an optical disk such as a CD, a DVD, and a BD. The semiconductor storage medium includes a read only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory.

[0030] The flowchart in FIG. 3 shows the process executed by the processor 23 by acquiring successive frames of a video. The processor 23 of the object tracking device 20 tracks the position of an object image 42 (see FIG. 4) every time a frame of a video is acquired according to the flowchart in FIG. 3. As shown in FIG. 2, there may be a plurality of objects 40 to be detected, and may include a pedestrian 40A, an automobile 40B, and a bicycle 40C. Furthermore, the object 40 is not limited to moving objects and people, and may include various targets such as obstacles on a road. In the following description of the object tracking method, one of the plurality of objects 40 (specifically, a pedestrian 40A) included in the video of the imaging device 10 installed behind the vehicle 100 will be used for the description. The other objects 40 (for example, an automobile 40B and a bicycle 40C) are also tracked by the same process.

[0031] The processor 23 acquires each frame of a moving image from the imaging device 10 via the input interface 21 (step S101). An example of one frame of a moving image is shown in FIG. 4. In the example of FIG. 4, an image of an object 40 (object image 42) behind the vehicle 100 is displayed in a two-dimensional image space 41 formed of a uv coordinate system. The u coordinate is the horizontal coordinate of the image. The v coordinate is the vertical coordinate of the image. In FIG. 4, the origin of the uv coordinate is the upper left corner point of the image space 41. The positive direction of the u coordinate is from left to right. The positive direction of the v coordinate is from top to bottom.

[0032] The processor 23 recognizes the image 42 of the object from each frame of the video by image recognition (step S102). Methods for recognizing the image 42 of the object include various known methods. For example, methods for recognizing the image 42 of the object include a method that uses shape recognition of objects such as cars and people, a template matching method, and a method that calculates features from an image and uses them for matching. A function approximator capable of learning the relationship between input and output can be used to calculate the features. A neural network, for example, can be used as the function approximator that can learn the relationship between input and output.

[0033] The processor 23 maps and transforms the coordinates (u, v) of the image 42 of the object in the image space 41 into coordinates (x', y') of the object in the virtual space 46 (see FIG. 6) (step S103). In general, the coordinates (u, v) of the image space 41, which are two-dimensional coordinates, cannot be transformed into coordinates (x, y, z) of the real space. However, by specifying the height in the real space and fixing the z coordinate to a predetermined value, the coordinates (u, v) of the image space 41 can be transformed into coordinates (x, y, z) of the real space. 0 )(z 0 It is possible to map the coordinates (x', y') of a two-dimensional virtual space 46 corresponding to the input information (x, y are fixed values). In this embodiment, the virtual space 46 is two-dimensional, but it may be three-dimensional depending on the input information (type of sensor).

[0034] 4, a representative point 43 located at the center of the bottom of the object image 42 is identified. For example, the representative point 43 can be the lowest position of the v coordinate and the center position of the range of the u coordinate of the area occupied by the object image 42 in the image space 41. This representative point 43 is assumed to be a position of the object 40 corresponding to the object image 42 that is in contact with the road surface or ground.

[0035] FIG. 5 shows the relationship between an object 40 located in a three-dimensional real space and an image 42 of the object in a two-dimensional image space 41. When the internal parameters of the imaging device 10 are known, the direction from the center of the imaging optical system 11 of the imaging device 10 to the corresponding coordinates (x, y, z) in the real space can be calculated based on the coordinates (u, v) in the image space 41. The internal parameters of the imaging device 10 include information such as the focal length and distortion of the imaging optical system 11 and the pixel size of the imaging element 12. In the real space, a point where a straight line directed toward a direction corresponding to a representative point 43 in the image space 41 intersects with a reference plane 44 of z=0 is defined as a mass point 45 of the object 40. The reference plane 44 corresponds to the road surface or ground on which the vehicle 100 is located. The mass point 45 has three-dimensional coordinates (x, y, 0). Therefore, when the two-dimensional space of z=0 is defined as a virtual space 46, the coordinates of the mass point 45 can be expressed as (x', y'). The coordinates (x', y') of mass point 45 in virtual space 46 correspond to the coordinates (x, y) of a specific point of object 40 on the xy plane (z=0) when object 40 is viewed from the direction along the z-axis in real space. The specific point is a point that corresponds to mass point 45.

[0036] As shown in FIG. 6, the processor 23 calculates the position (x', y') and velocity (v x´ ,v y´ ) is tracked (step S104). x´ ,v y´By having information on the position (x', y') of mass point 45 in the successive frames, processor 23 can predict the range of the position (x', y') of mass point 45 in the successive frames. Processor 23 can recognize mass point 45 located in the predicted range in the next frame as mass point 45 corresponding to image 42 of the tracked object. Each time a new frame is input, processor 23 sequentially calculates the position (x', y') and velocity (v x´ ,v y´ ) to update the

[0037] For tracking the mass point 45, for example, estimation using a Kalman filter based on a state space model can be adopted. By performing prediction / estimation using a Kalman filter, robustness against failure to detect and false detection of the object 40 to be detected is improved. In general, it is difficult to describe the image 42 of the object in the image space 41 with an appropriate model that describes the motion. Therefore, it has been difficult to easily estimate the position of the image 42 of the object in the image space 41 with high accuracy. In the object tracking device 20 of the present disclosure, the image 42 of the object is converted into a mass point 45 in the real space, which makes it possible to apply a model that describes the motion in the real space, thereby improving the accuracy of tracking the image 42 of the object. In addition, by treating the object 40 as a mass point 45 that does not have a size, simple and easy tracking is possible.

[0038] Each time the processor 23 estimates a new position of the mass point 45, the processor 23 may map and transform the coordinates of the mass point 45 in the virtual space 46 into coordinates (u, v) in the image space 41 to indicate the estimated position (step S105). The mass point 45 located at coordinates (x', y') in the virtual space 46 can be mapped and transformed into the image space 41 as a point located at coordinates (x, y, 0) in the real space. The coordinates (x, y, 0) in the real space can be mapped into coordinates (u, v) in the image space 41 of the imaging device 10 by a known method. The processor 23 can convert between the coordinates (u, v) in the image space 41, the coordinates (x', y') in the virtual space 46, and the coordinates (x, y, 0) in the real space.

[0039] (Data Association) FIG. 7 is a diagram for explaining data association. Data association is a process of associating a Kalman filter with an observation value. In data association, a plurality of Kalman filters can be associated with a plurality of observation values. Here, the observation value is the position of the detection target. The processor 23 assigns identifiers to the plurality of observation values ​​and the plurality of Kalman filters to distinguish them. In this embodiment, the processor 23 uses, for example, serial numbers to name the plurality of observation values ​​as observation value (1), observation value (2), observation value (3), etc. The processor 23 also uses, for example, symbols and serial numbers to name the plurality of Kalman filters as KF(1), KF(2), KF(3), etc.

[0040] In this embodiment, the processor 23 performs data association between M observations and N Kalman filters. M is an integer equal to or greater than 2. N is an integer equal to or greater than M. In the example of FIG. 7, the processor 23 performs data association between three observations and five Kalman filters. Observation (1) is the position of a pedestrian 40A detected in frame (k) of the video. Observation (2) is the position of a car 40B detected in frame (k) of the video. Observation (3) is the position of a bicycle 40C detected in frame (k) of the video. Frame (k-1) is the frame immediately preceding frame (k) in the video. Frame (k-2) is the frame immediately preceding frame (k) in the video. The current frame is assumed to be frame (k).

[0041] Here, KF(2) was used to track pedestrian 40A up until frame (k-1), but was initialized midway and is no longer used to track the position of the detection target. KF(5) is a new Kalman filter that was prepared when a new bicycle 40C was recognized in frame (k-2). KF(5) has started tracking the detection target because the newly recognized bicycle 40C was also recognized in the current frame (k). The other Kalman filters have been continuing to track the detection targets since frame (k-2).

[0042] In the example of FIG. 7, the processor 23 associates KF(1) with the observation value (1). The processor 23 associates KF(3) and KF(4) with the observation value (2). The processor 23 also associates KF(5) with the observation value (3). As in the example of the observation value (2), the processor 23 allows overlapping of detection results in the process of tracking multiple detection targets. That is, the processor 23 uses KF(3) and KF(4) to predict the range of the observation value (2), i.e., the position of the automobile 40B. In this way, by allowing overlapping in the data association, local optimization can be performed. For example, a method (e.g., the Hungarian method) in which multiple observation values ​​are associated one-to-one with multiple Kalman filters without allowing overlapping may cause a chain of misassociations due to global optimization. In this embodiment, since overlapping is allowed, problems such as chaining of misassociations do not occur. Furthermore, in the tracking process, one or more Kalman filters are associated with one observed value, making it difficult for tracking failures to occur for any observed value, thereby improving robustness.

[0043] (Tracked object ID management) Here, as described above, a plurality of Kalman filters may be associated with one observation value, but a plurality of observation values ​​may also be associated with one object to be detected. For example, when the detection object is an automobile 40B, and the automobile 40B disappears from the video due to a lane change or the like and then reappears in the video, a new observation value may be associated with the automobile 40B as a different object. In order to accurately track objects, it is preferable that the object tracking device 20 identifies each tracked object and grasps the correspondence with the observation value. In this embodiment, the processor 23 executes tracked object ID management using a hierarchical structure as described below, and groups a plurality of Kalman filters to determine whether they correspond to the same object.

[0044] FIG. 8 is a diagram showing a hierarchical structure of tracked object ID management (ID management) in this embodiment. Tracked object ID management is a process of associating a Kalman filter with a detection target. As shown in FIG. 8, the processor 23 manages an observation value, a Kalman filter, and a tracked object ID in each layer. The processor 23 also enables accurate tracking of an object by associating an observation value, a Kalman filter, and a tracked object ID. Here, the tracked object ID is unique identification information of a tracked object as described above. If the tracked object ID associated with multiple observation values ​​or multiple Kalman filters is the same, these observation values ​​or Kalman filters are related to tracking of the same object.

[0045] When a video frame is acquired, the processor 23 executes grouping of a plurality of Kalman filters. Then, the processor 23 updates the correspondence between the observation value, the Kalman filter, and the tracked object ID. In the example of FIG. 8, the processor 23 groups KF(1), KF(2), and KF(3), assigns an identifier "tracked object ID(1)" to the object to be tracked using these Kalman filters, and performs tracking control of the object. The processor 23 also groups KF(4) and KF(5), assigns an identifier "tracked object ID(2)" to the object to be tracked using these Kalman filters, and performs tracking control of the object. The processor 23 links the Kalman filters corresponding to the objects determined to be the same, and controls tracking in a hierarchical structure that also links the detection results of the detection targets corresponding to these Kalman filters, thereby enabling error-free and highly accurate tracking. The processor 23 can obtain a detection result with a high degree of certainty by, for example, comparing or selecting detection results using the linked Kalman filters.

[0046] (Limits on the fluctuation range of indices) As described above, a plurality of Kalman filters may be associated with one observation value, and a plurality of Kalman filters may be associated with one detection target (a detection target having one tracked object ID). By associating a plurality of Kalman filters, tracking failure is less likely to occur, and the tracking accuracy can be improved, thereby improving robustness. In addition, for stable tracking, the processor 23 may set a limit on the variation range of the index of the Kalman filter that affects the tracking of the detection target. As a specific example, the limit on the variation range of the index is to set at least one of an upper limit and a lower limit, which can be called a clipping process. In this embodiment, the index includes at least one of the Mahalanobis distance used to associate the Kalman filter with the observation value, the radius of the grouping region used to associate the Kalman filter with the detection target, and the size of the error ellipse of the Kalman filter. The limit on the variation range of each of these indexes will be described below.

[0047] The processor 23 may limit the range of variation of the index in the above data association. In this case, the index includes the Mahalanobis distance. The Mahalanobis distance represents the deviation of data, and in this embodiment, represents the deviation between the center of the error ellipse of the Kalman filter and the observed value. Here, the error ellipse of the Kalman filter represents an estimated range based on the probability density distribution of the position, and indicates that the position is located inside the ellipse with a predetermined probability (99% as an example). The error ellipse is calculated using the standard deviation in the x' direction and the standard deviation in the y' direction of the two-dimensional virtual space 46 (see FIG. 6), etc.

[0048] FIG. 9 is a diagram for explaining the restriction of the Mahalanobis distance. When there is no restriction on the Mahalanobis distance, if an observation value is included in an error ellipse, it is associated with a Kalman filter having that error ellipse. When a plurality of observation values ​​are included in the validation gate (boundary) of the error ellipse, the observation value having the smallest Mahalanobis distance from the center of the error ellipse is selected and associated with a Kalman filter having that error ellipse. Here, as the process of tracking the detection target progresses, the accuracy of tracking (confidence of the predicted position) improves and the size of the error ellipse becomes smaller. On the other hand, the position of the observation value may deviate from the center of the error ellipse of the Kalman filter due to the influence of measurement errors and noise of the imaging optical system 11. Therefore, as in the example of FIG. 9, when the size of the error ellipse becomes smaller and the observation value is no longer included in the validation gate, tracking will not continue if there is no restriction on the Mahalanobis distance. In this embodiment, the processor 23 sets a lower limit (see the dashed line in FIG. 9) on the Mahalanobis distance in the data association. When the validation gate of the error ellipse of the Kalman filter is inside the lower limit of the Mahalanobis distance, the processor 23 uses this lower limit of the Mahalanobis distance to associate the observed value with the Kalman filter, so that the association between the observed value and the Kalman filter is not lost and tracking continues.

[0049] Here, the lower limit of the Mahalanobis distance (see the dashed line in Fig. 9) is used only in matching the observed value with the Kalman filter in data association. In tracking the position of the detection target corresponding to the observed value (prediction calculation of the position), the original error ellipse (see the solid line in Fig. 9) is used in the calculation, so the tracking accuracy does not decrease.

[0050] Moreover, the processor 23 may limit the range of variation of the index in the above-mentioned tracking object ID management. Here, in the tracking object ID management, the association between the same detection target and the Kalman filter is performed by clustering such as DBSCAN (density-based spatial clustering of applications with noise). When the centers of the error ellipses of multiple Kalman filters are included in a grouping area of ​​a predetermined range, the processor 23 determines that these Kalman filters belong to one group. The clustering method is not limited to DBSCAN. For example, the clustering may be performed by another method such as the k-means method.

[0051] 10 and 11 are diagrams for explaining the restriction of the grouping area. In the examples of Fig. 10 and Fig. 11, the grouping area is shown as a circle. The index by which the processor 23 restricts the variation range includes the radius (eps) of the grouping area. The processor 23 sets at least one of an upper limit and a lower limit to the radius (eps) of the grouping area according to at least one of the observation noise and the distance to the detection target.

[0052] When the distance to the detection target is long, the range of the same detection target is easily affected by observation noise. Observation noise is, for example, image distortion caused by the lens of the imaging optical system 11. When the distance to the detection target is long, the processor 23 sets the radius (eps) of the grouping area to a large value as shown in FIG. 11 without exceeding the upper limit. The processor 23 sets an upper limit based on the distance dependency of the observation noise so that the range of the group of Kalman filters associated with the same detection target does not expand without limit. In the example of FIG. 11, three Kalman filters KF(p), KF(q), and KF(r) are associated with the same detection target.

[0053] When the distance to the detection target is short, the range of the same detection target is relatively less susceptible to the influence of observation noise. When the distance to the detection target is short, the processor 23 sets the radius (eps) of the grouping area to be small as shown in FIG. 10 within a range not falling below the lower limit. The processor 23 sets a lower limit so that the Kalman filter associated with the same detection target does not become zero based on the distance dependency of the observation noise. In the example of FIG. 10, two Kalman filters KF(p) and KF(r) are associated with the same detection target. In other words, KF(q) is associated with another detection target. For example, when tracking an approaching object, the processor 23 may gradually reduce the radius (eps) of the grouping area within a range from the upper limit to the lower limit. At this time, the processor 23 can determine that what was determined to be one object when it was far away (see FIG. 11) is two or more objects that are close to each other as it approaches (see FIG. 10). As another example, the processor 23 may variably set the radius of the grouping area depending only on the influence of the observation noise. For example, the processor 23 may adjust the radius of the grouping region in response to a change in observation noise due to the shooting environment (weather, for example).

[0054] In this way, the processor 23 can continue tracking with higher accuracy by varying the radius (eps) of the grouping area by setting at least one of an upper limit and a lower limit depending on at least one of the observation noise and the distance to the detection target.

[0055] Furthermore, the processor 23 may limit the range of variation of the index so that the error ellipse does not become too small with respect to the output accuracy (guaranteed accuracy) required for the detection result of the detection target. Here, the guaranteed accuracy may be set as, for example, an allowable error range. As described above in the limiting the range of variation of the index in data association, as the process of tracking the detection target progresses, the tracking accuracy (confidence of the predicted position) improves and the size of the error ellipse becomes smaller. On the other hand, if the size of the error ellipse becomes smaller and the observed value is no longer included in the validation gate, tracking may not be continued.

[0056] FIG. 12 is a diagram for explaining the limitation of the size of the error ellipse according to the guaranteed accuracy. In the example of FIG. 12, when frame (k-1) is changed to frame (k), the error ellipse of the Kalman filter becomes smaller than the guaranteed accuracy. Therefore, the result of the tracking calculation using the error ellipse of the Kalman filter of frame (k) is over-specified. In addition, since the error ellipse of the Kalman filter becomes smaller, the observed value may not be included in the validation gate. In this embodiment, the index by which the processor 23 limits the variation range includes the size of the error ellipse. The processor 23 sets a lower limit for the size of the error ellipse according to the guaranteed accuracy of the detection result of the detection target. In other words, the processor 23 adjusts the size of the error ellipse of the Kalman filter so that it does not fall below the guaranteed accuracy (see the solid line in FIG. 12) so that the correspondence between the observed value and the Kalman filter is continued.

[0057] Furthermore, the processor 23 may limit the range of variation of the index so that the size of the error ellipse is appropriate according to the saturation of the observation value due to the distance to the detection target. Here, the saturation of the observation value means that there is a limit to the improvement of the accuracy of the position of the observation value even if it is close, and the accuracy degradation does not change even if it is far away.

[0058] FIG. 13 is a diagram for explaining the limitation of the size of the error ellipse according to the saturation of the observation value. In the example of FIG. 13, the pedestrian 40A is sufficiently close to the vehicle 100 equipped with the object tracking device 20, and even if the error ellipse associated with the observation value of the pedestrian 40A becomes smaller, the detection accuracy does not improve due to the saturation of the observation value. In addition, in the example of FIG. 13, the automobile 40B is sufficiently far from the vehicle 100 equipped with the object tracking device 20, and even if the error ellipse associated with the observation value of the automobile 40B becomes larger, the detection accuracy does not change due to the saturation of the observation value. In the example of FIG. 13, the index by which the processor 23 limits the variation range includes the size of the error ellipse. The processor 23 sets at least one of the upper limit and the lower limit according to the saturation of the observation value due to the distance to the detection target. That is, in this embodiment, the processor 23 does not change the size of the error ellipse when the accuracy does not improve or does not decrease even if the size of the error ellipse is changed. This maintains the correspondence between the observation value and the Kalman filter, so that highly accurate tracking can be continued.

[0059] The processor 23 does not have to simultaneously execute all of the limitations on the variation ranges of the above-mentioned indices. In other words, the processor 23 may select some of the limitations on the variation ranges of the above-mentioned indices or combine them. The processor 23 may select only the Mahalanobis distance as an index for limiting the variation range, for example. The processor 23 may also select the radius of the grouping area and the size of the error ellipse according to the saturation of the observed value as an index for limiting the variation range, for example. In this case, the processor 23 may set only a lower limit or only an upper limit for the size of the error ellipse.

[0060] As described above, the object tracking device 20 according to this embodiment allows overlapping of detection results in the process of tracking multiple detection targets by the above configuration. Therefore, the object tracking device 20 can track multiple objects with high accuracy without causing a chain of misassociations. In addition, the object tracking device 20 according to this embodiment also restricts the range of fluctuation of the index. Therefore, the object tracking device 20 can stably track objects with high accuracy.

[0061] Although the embodiments according to the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions included in each component or each step can be rearranged so as not to be logically contradictory, and multiple components or steps can be combined into one or divided. Although the embodiments according to the present disclosure have been described mainly with respect to the device, the embodiments according to the present disclosure can also be realized as a method including steps executed by each component of the device. The embodiments according to the present disclosure can also be realized as a method, a program, or a storage medium on which a program is recorded, executed by a processor provided in the device. It should be understood that these are also included in the scope of the present disclosure. For example, the restriction of the range of variation of the index can also be realized as a method by including each step of the processing executed by the processor 23 in the object tracking method of FIG. 3.

[0062] In the above embodiment, the object tracking system 1 includes the imaging device 10, the object tracking device 20, and the display 30, but at least two of these may be integrated. For example, the function of the object tracking device 20 may be mounted on the imaging device 10. In this case, the imaging device 10 may include the above-mentioned storage unit 22 and output interface 24 in addition to the imaging optical system 11, the imaging element 12, and the processor 13. Furthermore, the processor 13 may execute the processing performed by the processor 23 in the above embodiment on the moving image output by the imaging device 10. With such a configuration, the imaging device 10 that performs object tracking may be realized.

[0063] In the present disclosure, the term "mobile body" includes vehicles, ships, and aircraft. In the present disclosure, the term "vehicle" includes, but is not limited to, automobiles and industrial vehicles, and may include railroad vehicles, lifestyle vehicles, and fixed-wing aircraft that run on runways. Automobiles include, but are not limited to, passenger cars, trucks, buses, motorcycles, trolley buses, and the like, and may include other vehicles that run on roads. Industrial vehicles include industrial vehicles for agriculture and construction. Industrial vehicles include, but are not limited to, forklifts and golf carts. Industrial vehicles for agriculture include, but are not limited to, tractors, cultivators, transplanters, binders, combines, and lawnmowers. Industrial vehicles for construction include, but are not limited to, bulldozers, scrapers, excavators, cranes, dump trucks, and road rollers. Vehicles include those that run by human power. Here, the classification of vehicles is not limited to the above. For example, automobiles may include industrial vehicles that can run on roads, and the same vehicle may be included in multiple classifications. In the present disclosure, the vessel includes a marine jet, a boat, and a tanker. In the present disclosure, the aircraft includes a fixed-wing aircraft and a rotorcraft. [Explanation of symbols]

[0064] 1. Object Tracking System 10. Imaging device 11 Imaging optical system 12 Image sensor 13 Processors 20 Object Tracking Device 21 Input Interface 22 Memory section 23 Processors 24 Output Interfaces 30 Display 40 Object 40A Pedestrian 40B car 40C Bicycle 41 Image Space 42 Image of an object 43 Representative points 44 Reference plane 45 mass points 46 Virtual Space 100 vehicles

Claims

1. an input interface for acquiring sensor data; a processor that detects a detection target from the sensor data and tracks the detection target using a Kalman filter in which the detection target and an observation value indicating a position of the detection target are associated with each other; an output interface for outputting a detection result of the detection object; The processor sets a limit on a range of variation of an index of the Kalman filter that affects tracking of the detection target.

2. 2. The object tracking device according to claim 1, wherein the index includes at least one of a Mahalanobis distance used to match the Kalman filter with the observation value, a radius of a grouping area used to match the Kalman filter with the detection target, and a size of an error ellipse of the Kalman filter.

3. the index includes the Mahalanobis distance, The object tracking device according to claim 2 , wherein the processor sets a lower limit on the Mahalanobis distance.

4. The indicator includes a radius of the grouping area, The object tracking device according to claim 2 , wherein the processor sets at least one of an upper limit and a lower limit for the radius of the grouping region depending on at least one of an observation noise and a distance to the detection target.

5. The index includes the size of the error ellipse, The object tracking device according to claim 2 , wherein the processor sets a lower limit on the size of the error ellipse depending on a guaranteed accuracy of the detection result of the detection target.

6. The index includes the size of the error ellipse, The object tracking device according to claim 2 , wherein the processor sets at least one of an upper limit and a lower limit on the size of the error ellipse depending on saturation of the observation value due to the distance to the detection target.

Images