Object tracking device and object tracking method

The system improves object tracking accuracy by using multiple Kalman filters with overlap detection and hierarchical management to track multiple objects effectively, addressing the challenges of conventional systems.

JP7583827B2Active Publication Date: 2024-11-14KYOCERA CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2022558899
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-10-28
Filing Date
2021-09-09
Publication Date
2024-11-14
Estimated Expiration
2041-09-09

AI Technical Summary

Technical Problem

Existing object tracking systems face challenges in accurately tracking multiple detection targets due to overlapping images, leading to incorrect tracking or reduced accuracy, particularly when using conventional techniques with Kalman filters.

Method used

The system employs a Kalman filter for each detection target, allowing overlap of detection results by associating multiple Kalman filters with each target, managing observed values, and using a hierarchical structure to determine identical objects, thereby improving tracking accuracy.

Benefits of technology

This approach enhances the precision of tracking multiple objects by preventing chain misassociations and ensuring robustness, even in situations with false detections or temporary object disappearance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007583827000001
    Figure 0007583827000001
  • Figure 0007583827000002
    Figure 0007583827000002
  • Figure 0007583827000003
    Figure 0007583827000003
Patent Text Reader

Abstract

Provided are an object tracking device and an object tracking method with which it is possible to track a plurality of objects with high accuracy. An object tracking device (20) comprises: an input interface (21) that acquires sensor data; a processor (23) that detects a plurality of objects to be detected from the sensor data, and tracks each of the plurality of objects to be detected using Kalman filtering; and an output interface (24) that outputs the detection results of the objects to be detected. The processor (23) allows overlapping of detection results during the process of tracking the plurality of objects to be detected.
Need to check novelty before this filing date? Find Prior Art

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Japanese Patent Application No. 2020-180783 (filed October 28, 2020), the entire disclosure of which is incorporated herein by reference. [Technical field]

[0002] The present disclosure relates to an object tracking device and an object tracking method. [Background technology]

[0003] There are known techniques for detecting surrounding objects, tracking the detected objects, and predicting their movements. For example, Patent Document 1 discloses a device that processes video signals output from an on-board camera that captures video around a vehicle, detects the presence or absence of approaching vehicles and pedestrians, and displays the approaching vehicles and pedestrians by adding square frame marks. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 11-321494 Summary of the Invention

[0005] An object tracking device according to an embodiment includes: an input interface for acquiring sensor data; a processor that detects a plurality of detection targets from the sensor data and tracks each of the plurality of detection targets using a Kalman filter; an output interface for outputting a detection result of the detection object; The processor allows overlapping of detection results in the process of tracking the multiple detection targets.

[0006] An object tracking method according to an embodiment includes: Acquiring sensor data; detecting a plurality of detection targets from the sensor data, and tracking each of the plurality of detection targets using a Kalman filter; and outputting a detection result of the detection target. The tracking allows for overlapping of detection results in the process of tracking the multiple detection targets.

[0007] An object tracking device according to an embodiment includes: an input interface for acquiring a plurality of sensor data obtained by different sensing methods; a processor that detects a plurality of detection targets from the plurality of sensor data and performs data processing to track each of the plurality of detection targets using a Kalman filter; The processor allows detection results of the plurality of pieces of sensor data to be redundantly linked to one of the plurality of detection targets.

[0008] An object tracking method according to an embodiment includes: Acquiring a plurality of sensor data obtained by different sensing methods; detecting a plurality of detection targets from the plurality of sensor data, and performing data processing for tracking each of the plurality of detection targets using a Kalman filter; The data processing allows the detection results of the plurality of sensor data to be redundantly linked to one of the plurality of detection targets. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of an object tracking system including an object tracking device according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing a vehicle equipped with the object tracking system of FIG. 1 and a detection target. [Diagram 3] FIG. 3 is a flowchart showing an example of a process for tracking an image of an object on a moving image. [Figure 4]FIG. 4 is a diagram showing an example of an image of an object on a moving image. [Diagram 5] FIG. 5 is a diagram for explaining the relationship between an object in real space, an image of the object in a moving image, and a mass point in virtual space. [Figure 6] FIG. 6 is a diagram showing an example of movement of a mass point in a virtual space. [Figure 7] FIG. 7 is a diagram for explaining the operation modes of the Kalman filter. [Figure 8] FIG. 8 is a diagram for explaining data association. [Figure 9] FIG. 9 is a diagram for explaining the representative Kalman filter. [Figure 10] FIG. 10 is a diagram illustrating a layer configuration for tracking object ID management. [Figure 11] FIG. 11 is a diagram for explaining the determination of the same object. [Figure 12] FIG. 12 is a diagram for explaining inheritance of a tracked object ID. [Figure 13] FIG. 13 is a diagram for explaining fusion. [Figure 14] FIG. 14 is a diagram for explaining overlapping application of fusion. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The drawings used in the following description are schematic. The dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0011] Fig. 1 is a block diagram showing a schematic configuration of an object tracking system 1. An object tracking device 20 according to an embodiment of the present disclosure is included in the object tracking system 1. In this embodiment, the object tracking system 1 includes an imaging device 10, the object tracking device 20, and a display 30. Moreover, the object tracking system 1 is mounted on a vehicle 100, which is an example of a moving body, as illustrated in Fig. 2.

[0012] The object tracking device 20 according to this embodiment acquires a moving image from the imaging device 10 as sensor data. That is, in this embodiment, the sensor used to detect multiple detection targets is the imaging element 12 that captures visible light provided in the imaging device 10. However, the object tracking system 1 is not limited to the configuration shown in FIG. 1. The object tracking system 1 can be equipped with a device different from the imaging device 10 as long as it detects multiple detection targets. As another example, the object tracking system 1 may be configured to include a measuring device that measures the distance to the detection target from the reflected wave of the irradiated laser light instead of the imaging device 10. As another example, the object tracking system 1 may be configured to include a detection device having a millimeter wave sensor instead of the imaging device 10. Also, as another example, the object tracking system 1 may be configured to include an imaging device 10 equipped with an imaging element 12 that captures light other than the visible light region. In addition, the object tracking system 1 may be configured to include at least one of an imaging device 10 that detects visible light, a measuring device that measures the distance to the detection target from the reflected waves of irradiated laser light, a detection device having a millimeter wave sensor, and an imaging device 10 that detects light other than visible light.

[0013] In this embodiment, the object tracking system 1 is mounted on a moving body and detects objects 40 (see FIG. 2) around the moving moving body. However, the object tracking system 1 is not limited to being mounted on a moving body. As another example, the object tracking system 1 may be used in a facility such as a factory and detect employees, transport robots, and products. As another example, the object tracking system 1 may be used in an elderly welfare facility and detect elderly people and staff in the room. The object tracking system 1 may not only track objects for the safety of driving or activities, but also track objects for the efficiency of work, quality control, or productivity improvement in agricultural and industrial sites. Here, in the present disclosure, objects that are detection targets of the object tracking device 20 include not only objects such as moving bodies but also people.

[0014] As shown in FIG. 2, in this embodiment, the x-axis direction of the coordinates of the real space is the width direction of the vehicle 100 on which the imaging device 10 is installed. The y-axis direction is the direction in which the vehicle 100 moves backward. The x-axis and y-axis directions are parallel to the road surface on which the vehicle 100 is located. The z-axis direction is perpendicular to the road surface. The z-axis direction can be called the vertical direction. The x-axis, y-axis, and z-axis directions are mutually orthogonal. The way in which the x-axis, y-axis, and z-axis directions are taken is not limited to this. The x-axis, y-axis, and z-axis directions can be interchanged with each other.

[0015] The imaging device 10 includes an imaging optical system 11, an imaging element 12, and a processor 13.

[0016] The imaging device 10 may be installed at various positions on the vehicle 100. The imaging device 10 includes, but is not limited to, a front camera, a left side camera, a right side camera, and a rear camera. The front camera, the left side camera, the right side camera, and the rear camera are installed on the vehicle 100 so as to be able to capture images of the surrounding areas in front of, on the left side of, on the right side of, and behind the vehicle 100, respectively. In an embodiment described below as an example, as shown in FIG. 2, the imaging device 10 is attached to the vehicle 100 with its optical axis direction facing downward from the horizontal direction so as to be able to capture images of the rear of the vehicle 100.

[0017] The imaging optical system 11 may include one or more lenses. The imaging element 12 may include a charge-coupled device image sensor (CCD image sensor) or a complementary MOS image sensor (CMOS image sensor).

[0018] The imaging element 12 converts an image of an object (subject image) formed on the imaging surface of the imaging element 12 by the imaging optical system 11 into an electrical signal. The imaging element 12 can capture moving images at a predetermined frame rate. A frame is each still image that constitutes a moving image. The number of images that can be captured per second is called the frame rate. The frame rate may be, for example, 60 fps (frames per second) or 30 fps.

[0019] The processor 13 controls the entire imaging device 10 and performs various image processing on the moving images output from the imaging element 12. The image processing performed by the processor 13 may include any processing such as distortion correction, brightness adjustment, contrast adjustment, gamma correction, etc.

[0020] The processor 13 may be comprised of one or more processors. The processor 13 includes one or more circuits or units configured to perform one or more data computing procedures or processes, for example, by executing instructions stored in an associated memory. The processor 13 includes one or more processors, microprocessors, microcontrollers, application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), or any combination of these devices or configurations or other known devices or configurations.

[0021] The object tracking device 20 includes an input interface 21, a memory unit 22, a processor 23, and an output interface 24.

[0022] The input interface 21 is configured to be able to communicate with the imaging device 10 via a wired or wireless communication means. The input interface 21 acquires moving images from the imaging device 10 as sensor data. The input interface 21 may correspond to a transmission method of an image signal transmitted by the imaging device 10. The input interface 21 can be referred to as an input unit or an acquisition unit. The imaging device 10 and the input interface 21 may be connected via an in-vehicle communication network such as a control area network (CAN).

[0023] The storage unit 22 is a storage device that stores data and programs necessary for processing performed by the processor 23. For example, the storage unit 22 temporarily stores moving images acquired from the imaging device 10. For example, the storage unit 22 stores data generated by processing performed by the processor 23. The storage unit 22 may be configured using, for example, one or more of a semiconductor memory, a magnetic memory, an optical memory, and the like. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk and a magnetic tape. The optical memory may include, for example, a compact disc (CD), a digital versatile disc (DVD), and a Blu-ray (registered trademark) disc (BD).

[0024] The processor 23 controls the entire object tracking device 20. The processor 23 recognizes an image of an object included in a moving image acquired via the input interface 21. The processor 23 performs mapping conversion of the coordinates of the recognized image of the object into the coordinates of the object 40 in a virtual space 46 (see FIG. 6), and tracks the position and velocity of a mass point 45 (see FIG. 5) representing the object 40 in the virtual space 46. The mass point 45 is a point that has mass and no size. The virtual space 46 is a two-dimensional space in which the value in the z-axis direction is a predetermined fixed value in a coordinate system consisting of three axes of the x-axis, y-axis, and z-axis in the real space. The processor 23 may perform mapping conversion of the coordinates of the tracked mass point 45 in the virtual space 46 into coordinates in the moving image.

[0025] Moreover, the processor 23 detects a plurality of detection targets from a moving image, and tracks each of the plurality of detection targets using a Kalman filter. When detecting a plurality of detection targets, if their images overlap in a moving image, tracking may be erroneous or accuracy may decrease in conventional techniques. In this embodiment, the processor 23 can avoid such problems by associating one or more Kalman filters with each of the plurality of detection targets. Moreover, the processor 23 manages the observation value, the Kalman filter, and the unique identification information of the tracked object (hereinafter, "tracked object ID") in each layer (hierarchy). The processor 23 determines whether the tracked object is the same object or not, and executes a process of associating the observation value, the Kalman filter, and the tracked object ID. This can further improve the accuracy of tracking the plurality of detection targets. Details of the process performed by the processor 23 will be described later. The processor 23 may include a plurality of processors, similar to the processor 13 of the imaging device 10. Moreover, the processor 23 may be configured by combining a plurality of types of devices, similar to the processor 13.

[0026] The output interface 24 is configured to output an output signal from the object tracking device 20. The output interface 24 may be referred to as an output unit. The output interface 24 may output a detection result of the detection target, such as the coordinates of the mass point 45.

[0027] The output interface 24 may include a physical connector and a wireless communication device. The output interface 24 may be connected to a network of the vehicle 100, such as a CAN. The output interface 24 may be connected to the display 30, a control device, an alarm device, and the like of the vehicle 100 via a communication network such as a CAN. The information output from the output interface 24 may be used as appropriate by each of the display 30, the control device, and the alarm device.

[0028] The display 30 may display a moving image output from the object tracking device 20. When the display 30 receives coordinates of a mass point 45 representing the position of an image of an object from the object tracking device 20, the display 30 may have a function of generating an image element (e.g., a warning to be displayed together with an approaching object) according to the coordinates and superimposing the image on the moving image. The display 30 may employ various types of devices. For example, the display 30 may employ a liquid crystal display (LCD), an organic electro-luminescence (EL) display, an inorganic EL display, a plasma display panel (PDP), a field emission display (FED), an electrophoretic display, a twist ball display, or the like.

[0029] Next, the object tracking method executed by the object tracking device 20 will be described in detail with reference to the flowchart of FIG. 3. The object tracking device 20 may be configured to read and implement the processing performed by the processor 23 described below, which is recorded in a non-transitory computer-readable medium. The non-transitory computer-readable medium includes, but is not limited to, a magnetic storage medium, an optical storage medium, a magneto-optical storage medium, and a semiconductor storage medium. The magnetic storage medium includes a magnetic disk, a hard disk, and a magnetic tape. The optical storage medium includes an optical disk such as a CD, a DVD, and a BD. The semiconductor storage medium includes a read only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory.

[0030] The flowchart in FIG. 3 shows the process executed by the processor 23 by acquiring successive frames of a video. The processor 23 of the object tracking device 20 tracks the position of an object image 42 (see FIG. 4) every time a frame of a video is acquired according to the flowchart in FIG. 3. As shown in FIG. 2, there are multiple objects 40 to be detected, and they may include a pedestrian 40A, an automobile 40B, and a bicycle 40C. Furthermore, the object 40 is not limited to moving objects and people, and may include various targets such as obstacles on a road. In the following description of the object tracking method, one of the multiple objects 40 (specifically, a pedestrian 40A) included in the video of the imaging device 10 installed behind the vehicle 100 will be used for the description. The other objects 40 (for example, an automobile 40B and a bicycle 40C) are also tracked by the same process.

[0031] The processor 23 acquires each frame of a moving image from the imaging device 10 via the input interface 21 (step S101). An example of one frame of a moving image is shown in FIG. 4. In the example of FIG. 4, an image of an object 40 (object image 42) behind the vehicle 100 is displayed in a two-dimensional image space 41 formed of a uv coordinate system. The u coordinate is the horizontal coordinate of the image. The v coordinate is the vertical coordinate of the image. In FIG. 4, the origin of the uv coordinate is the upper left corner point of the image space 41. The positive direction of the u coordinate is from left to right. The positive direction of the v coordinate is from top to bottom.

[0032] The processor 23 recognizes the image 42 of the object from each frame of the video by image recognition (step S102). Methods for recognizing the image 42 of the object include various known methods. For example, methods for recognizing the image 42 of the object include a method that uses shape recognition of objects such as cars and people, a template matching method, and a method that calculates features from an image and uses them for matching. A function approximator capable of learning the relationship between input and output can be used to calculate the features. A neural network, for example, can be used as the function approximator that can learn the relationship between input and output.

[0033] The processor 23 performs mapping conversion of the coordinates (u, v) of the image 42 of the object in the image space 41 to the coordinates (x', y') of the object in the virtual space 46 (see FIG. 6) (step S103). In general, the coordinates (u, v) of the image space 41, which are two-dimensional coordinates, cannot be converted to the coordinates (x, y, z) of the real space. However, by identifying the height in the real space and fixing the z coordinate to a predetermined value, it becomes possible to map the coordinates (u, v) of the image space 41 to the coordinates (x', y') of the two-dimensional virtual space 46 corresponding to the coordinates (x, y, z0) (z0 is a fixed value) of the real space. Here, although the virtual space 46 is two-dimensional in this embodiment, it may be three-dimensional depending on the input information (type of sensor).

[0034] 4, a representative point 43 located at the center of the bottom of the object image 42 is identified. For example, the representative point 43 can be the lowest position of the v coordinate and the center position of the range of the u coordinate of the area occupied by the object image 42 in the image space 41. This representative point 43 is assumed to be a position of the object 40 corresponding to the object image 42 that is in contact with the road surface or ground.

[0035] FIG. 5 shows the relationship between an object 40 located in a three-dimensional real space and an image 42 of the object in a two-dimensional image space 41. When the internal parameters of the imaging device 10 are known, the direction from the center of the imaging optical system 11 of the imaging device 10 to the corresponding coordinates (x, y, z) in the real space can be calculated based on the coordinates (u, v) in the image space 41. The internal parameters of the imaging device 10 include information such as the focal length and distortion of the imaging optical system 11 and the pixel size of the imaging element 12. In the real space, a point where a straight line directed toward a direction corresponding to a representative point 43 in the image space 41 intersects with a reference plane 44 of z=0 is defined as a mass point 45 of the object 40. The reference plane 44 corresponds to the road surface or ground on which the vehicle 100 is located. The mass point 45 has three-dimensional coordinates (x, y, 0). Therefore, when the two-dimensional space of z=0 is defined as a virtual space 46, the coordinates of the mass point 45 can be expressed as (x', y'). The coordinates (x', y') of mass point 45 in virtual space 46 correspond to the coordinates (x, y) of a specific point of object 40 on the xy plane (z=0) when object 40 is viewed from the direction along the z-axis in real space. The specific point is a point that corresponds to mass point 45.

[0036] As shown in FIG. 6, the processor 23 calculates the position (x', y') and velocity (v x´ ,v y´ ) is tracked (step S104). x´ ,v y´ By having information on the position (x', y') of mass point 45 in the successive frames, processor 23 can predict the range of the position (x', y') of mass point 45 in the successive frames. Processor 23 can recognize mass point 45 located in the predicted range in the next frame as mass point 45 corresponding to image 42 of the tracked object. Each time a new frame is input, processor 23 sequentially calculates the position (x', y') and velocity (v x´ ,v y´ ) to update the

[0037] For tracking the mass point 45, for example, estimation using a Kalman filter based on a state space model can be adopted. By performing prediction / estimation using a Kalman filter, robustness against failure to detect and false detection of the object 40 to be detected is improved. In general, it is difficult to describe the image 42 of the object in the image space 41 with an appropriate model that describes the motion. Therefore, it has been difficult to easily estimate the position of the image 42 of the object in the image space 41 with high accuracy. In the object tracking device 20 of the present disclosure, the image 42 of the object is converted into a mass point 45 in the real space, which makes it possible to apply a model that describes the motion in the real space, thereby improving the accuracy of tracking the image 42 of the object. In addition, by treating the object 40 as a mass point 45 that does not have a size, simple and easy tracking is possible.

[0038] Each time the processor 23 estimates a new position of the mass point 45, the processor 23 may map and transform the coordinates of the mass point 45 in the virtual space 46 into coordinates (u, v) in the image space 41 to indicate the estimated position (step S105). The mass point 45 located at coordinates (x', y') in the virtual space 46 can be mapped and transformed into the image space 41 as a point located at coordinates (x, y, 0) in the real space. The coordinates (x, y, 0) in the real space can be mapped into coordinates (u, v) in the image space 41 of the imaging device 10 by a known method. The processor 23 can convert between the coordinates (u, v) in the image space 41, the coordinates (x', y') in the virtual space 46, and the coordinates (x, y, 0) in the real space.

[0039] In this embodiment, the processor 23 detects a plurality of detection targets from a moving image and tracks each of them. For example, in a situation as shown in FIG. 2, the processor 23 tracks each of a pedestrian 40A, a car 40B, and a bicycle 40C. The processor 23 tracks the position and speed of a mass point 45 representing each of the plurality of detection targets using a virtual space 46. In order for the processor 23 to execute the object tracking method shown in FIG. 3, a Kalman filter must be prepared for each of the plurality of objects 40. In this embodiment, when the processor 23 recognizes an image 42 of a new object in a moving image, the processor 23 generates a new Kalman filter, and when a start condition is satisfied, the processor 23 performs tracking. As a result, one or more Kalman filters are prepared for each of the plurality of objects 40 that are detection targets. However, if Kalman filters are continuously generated for a transient new object 40 (for example, an oncoming vehicle that is not included in the moving image after a certain period of time), the number of Kalman filters increases more than necessary, the amount of calculation increases, and there is a risk of a processing delay in tracking the object. In this embodiment, the processor 23 initializes the Kalman filter when the abort condition is met, so as to avoid processing delays.

[0040] 7 is a diagram for explaining the operation modes of the Kalman filter. The processor 23 controls the Kalman filter according to the initial state, the tracking preparation state, and the tracking state of the detection target.

[0041] The initial state of the detection target is a state in which the processor 23 recognizes an image 42 of a new object to be detected in a moving image. At this time, the operating mode of the Kalman filter associated with the detection target is "mode 0". A Kalman filter in mode 0 does not have initial values ​​(position and velocity information). When the Kalman filter associated with the detection target is in mode 0, the processor 23 does not track the position of the detection target, i.e., does not predict the range of the position (x', y') of the mass point 45 in the next frame.

[0042] The tracking preparation state is a state in which an image 42 of an object newly recognized in the previous frame is also recognized in the current frame. At this time, the operation mode of the Kalman filter associated with the detection target is "mode 1". The Kalman filter in mode 1 obtains the position (x', y') of the mass point 45 of the detection target, but since there is no information on the position of the detection target in the previous frame, it does not obtain the velocity (v x´ ,v y´ ) is not acquired. In other words, the Kalman filter in mode 1 has only a part of the necessary initial values ​​(position and velocity information). When the Kalman filter associated with the detection target is in mode 1, the processor 23 does not track the position of the detection target.

[0043] When the Kalman filter is in mode 1, a process is performed to confirm that the object image 42 is not a false detection. As shown in Fig. 7, when the first termination condition is satisfied, that is, when it is determined to be a false detection or when the object image 42 disappears, the operation mode of the Kalman filter is initialized and returns to mode 0. This makes it possible to avoid starting tracking due to sporadic false detection.

[0044] Also, as shown in FIG. 7, when the starting condition is satisfied, that is, when the Kalman filter is in mode 1 and an object image 42 newly recognized in the frame two frames before is also recognized in the current frame, the operating mode of the Kalman filter becomes mode 2.

[0045] The tracking state is a state in which the second termination condition is not satisfied after the above-mentioned start condition is satisfied. The second termination condition is that the image 42 of the object disappears in a predetermined number of consecutive frames up to the current frame. At this time, the operation mode of the Kalman filter associated with the detection target is "mode 2". The Kalman filter in mode 2 has the necessary initial values ​​(position and velocity information) and can be immediately used for tracking control. When the Kalman filter associated with the detection target is in mode 2, the processor 23 tracks the position of the detection target.

[0046] As shown in FIG. 7, when the Kalman filter associated with the detection target is in mode 2 and the second termination condition is satisfied, the operation mode of the Kalman filter is initialized and returns to mode 0.

[0047] In this way, when the same detection target is detected consecutively, the processor 23 sets the Kalman filter to a tracking state (mode 2). Here, the number of consecutive detections is 2 in this embodiment, but it may be 3 or more. When the number of consecutive detections is 3 or more, for example, the state of mode 1 (tracking preparation state) may continue for a long time.

[0048] Moreover, the processor 23 stops tracking using the Kalman filter when the same detection target cannot be detected a predetermined number of times in succession. Here, the predetermined number is 5 in this embodiment, but is not limited to this. In tracking an object using the Kalman filter, even if there is no information on the position of the detection target obtained from a video image, it is possible to continue predicting the range of the position of the detection target. However, the error in the predicted range of the position increases as the number of frames in which such information cannot be obtained increases. The above-mentioned predetermined number may be determined based on the magnitude of this error.

[0049] The processor 23 sets the above-mentioned operation modes for the Kalman filters and dynamically changes the settings, thereby enabling systematic control for a plurality of Kalman filters according to their states.

[0050] FIG. 8 is a diagram for explaining data association. Data association is a correspondence between a plurality of observation values ​​and a plurality of Kalman filters. Here, the observation value is the position of the detection target. The processor 23 assigns identifiers to the plurality of observation values ​​and the plurality of Kalman filters to distinguish them. In this embodiment, the processor 23 uses, for example, serial numbers to name the plurality of observation values ​​as observation value (1), observation value (2), observation value (3), and so on. The processor 23 also uses, for example, symbols and serial numbers to name the plurality of Kalman filters as KF(1), KF(2), KF(3), and so on.

[0051] In this embodiment, the processor 23 performs data association between M observations and N Kalman filters. M is an integer equal to or greater than 2. N is an integer equal to or greater than M. In the example of FIG. 8, the processor 23 performs data association between three observations and five Kalman filters. Observation (1) is the position of a pedestrian 40A detected in frame (k) of the video. Observation (2) is the position of a car 40B detected in frame (k) of the video. Observation (3) is the position of a bicycle 40C detected in frame (k) of the video. Frame (k-1) is the frame immediately preceding frame (k) in the video. Frame (k-2) is the frame immediately preceding frame (k) in the video. The current frame is assumed to be frame (k).

[0052] Here, KF(2) was used to track pedestrian 40A up until frame (k-1), but was initialized because the second termination condition was satisfied. In other words, the operation mode of KF(2) is mode 0, and it is not used to track the position of the detection target. Also, KF(5) is a newly prepared Kalman filter because a new bicycle 40C was recognized in frame (k-2). KF(5) was in mode 1 in frame (k-1), but has become mode 2 because the start condition has been satisfied. The other Kalman filters have been in mode 2 since frame (k-2), and each continues to track the detection target.

[0053] In the example of FIG. 8, the processor 23 associates KF(1) with the observation value (1). The processor 23 associates KF(3) and KF(4) with the observation value (2). The processor 23 also associates KF(5) with the observation value (3). As in the example of the observation value (2), the processor 23 allows overlapping of detection results in the process of tracking multiple detection targets. That is, the processor 23 predicts the range of the observation value (2), i.e., the position of the automobile 40B, using KF(3) and KF(4). In this way, by allowing overlapping in the data association, local optimization can be performed. For example, a method (e.g., the Hungarian method) that does not allow overlapping and associates multiple observation values ​​with multiple Kalman filters one-to-one may cause a chain of misassociations due to global optimization. In this embodiment, overlapping is allowed, so there is no problem of chaining of misassociations. Furthermore, in the tracking process, one or more Kalman filters are associated with one observed value, making it difficult for tracking failures to occur for any observed value, thereby improving robustness.

[0054] A situation in which multiple Kalman filters are associated with one observation value may be, for example, when one object is recognized as two objects due to the influence of light reflection, and a new Kalman filter is associated with one of the objects. As described above, the tracking control of the detection target is executed in parallel using the associated multiple Kalman filters. However, for example, when the predicted position of the detection target is used to prevent collision of the vehicle 100, it may be preferable to output one detection result with the highest degree of certainty from the output interface 24. The processor 23 may determine the Kalman filter (hereinafter, "representative Kalman filter") that shows the detection result with the highest degree of certainty based on the error ellipse of the Kalman filter.

[0055] FIG. 9 is a diagram for explaining a representative Kalman filter. In the example of FIG. 9, three Kalman filters, KF(p), KF(q), and KF(r), are associated with one observation value. The processor 23 calculates an error ellipse for each of the three Kalman filters. The error ellipse indicates an estimated range based on a probability density distribution of a position, and indicates that the position is located inside the ellipse with a predetermined probability (for example, 99%). The error ellipse is calculated using a standard deviation in the x' direction and a standard deviation in the y' direction. The processor 23 determines the Kalman filter having the smallest error ellipse as the representative Kalman filter. In the example of FIG. 9, KF(r) is the representative Kalman filter.

[0056] In this way, when the detected multiple detection targets are considered to be the same object, the processor 23 can select the one with the smallest estimation range based on the probability density distribution of the positions of the multiple detection targets as the representative of the object. Therefore, the object tracking device 20 is also suitable for use in driving assistance such as collision prevention for the vehicle 100.

[0057] Here, as described above, a plurality of Kalman filters may be associated with one observation value, but a plurality of observation values ​​may also be associated with one object to be detected. For example, when the detection object is an automobile 40B, and the automobile 40B disappears from the video due to a lane change or the like and then reappears in the video, a new observation value may be associated with the automobile 40B as a different object. In order to accurately track objects, it is preferable that the object tracking device 20 identifies each tracked object and grasps the association with the observation value. In this embodiment, the processor 23 performs hierarchical management as described below, and groups a plurality of Kalman filters to determine whether they correspond to the same object.

[0058] Fig. 10 is a diagram showing a layer structure of tracking object ID management in this embodiment. As shown in Fig. 10, the processor 23 manages the observation value, the Kalman filter, and the tracking object ID in each layer. The processor 23 also enables accurate tracking of an object by associating the observation value, the Kalman filter, and the tracking object ID. Here, the tracking object ID is unique identification information of the tracking object as described above. If the tracking object ID associated with multiple observation values ​​or multiple Kalman filters is the same, these observation values ​​or Kalman filters are related to tracking of the same object.

[0059] As described above, the processor 23 generates a Kalman filter for a new observation value, and associates one or more Kalman filters with one observation value. The processor 23 further associates the Kalman filter with a tracking object ID. FIG. 11 is a diagram for explaining the determination of the same object. The processor 23 groups a plurality of Kalman filters by clustering such as DBSCAN (density-based spatial clustering of applications with noise). The processor 23 determines that the Kalman filters belong to one group when the centers of the error ellipses of the plurality of Kalman filters are included in a predetermined range. In the example of FIG. 11, the predetermined range is indicated by a circle. Moreover, KF(p), KF(q) and KF(r) are one group. Here, the predetermined range may change according to the size of the tracking object. For example, if the tracking object is a car 40B, the predetermined range may be set larger than when the tracking object is a pedestrian 40A. Moreover, the predetermined range may be constant regardless of the type of tracking object. The clustering method is not limited to DBSCAN. Clustering may be performed using other techniques, such as the k-means method.

[0060] When a video frame is acquired, the processor 23 executes grouping of a plurality of Kalman filters. Then, the processor 23 updates the correspondence between the observation value, the Kalman filter, and the tracked object ID. In the example of FIG. 10, the processor 23 groups KF(1), KF(2), and KF(3), assigns an identifier "tracked object ID(1)" to the object to be tracked using these Kalman filters, and performs tracking control of the object. The processor 23 also groups KF(4) and KF(5), assigns an identifier "tracked object ID(2)" to the object to be tracked using these Kalman filters, and performs tracking control of the object.

[0061] Here, KF(1) and KF(2) are associated with the observation value (1), and KF(3) is associated with the observation value (2). The processor 23 can identify that the observation value (1) and the observation value (2), which were determined to be the positions of different objects by grouping, are the positions of the same object with the tracking object ID (1) as an identifier. The processor 23 links the Kalman filters corresponding to the objects determined to be the same, and controls tracking in a hierarchical structure in which the detection results of the detection targets corresponding to these Kalman filters are also linked, thereby enabling error-free and highly accurate tracking. The processor 23 can obtain a detection result with a high degree of certainty by, for example, comparing or selecting detection results using a plurality of linked Kalman filters. In addition, the processor 23 can continue tracking the object with the tracking object ID (1) as an identifier by using the observation value (1), KF(1), and KF(2), even if, for example, the observation value (2) is lost or KF(3) is initialized. In other words, robustness can be improved.

[0062] Here, the processor 23 may determine the Kalman filter having the smallest error ellipse among the multiple Kalman filters belonging to the same group as the representative Kalman filter, as described above (see FIG. 9). In other words, the processor 23 may determine the one having the smallest estimation range based on the probability density distribution of the positions of the multiple detection targets among the Kalman filters corresponding to the same grouped object as the representative of the group.

[0063] 12 is a diagram for explaining the inheritance of the tracked object ID. As described above, the processor 23 executes grouping of a plurality of Kalman filters when a video frame is acquired. When the composition of the group changes, the processor 23 appropriately inherits the tracked object ID based on the similarity.

[0064] In the example of Fig. 12, when frame (k-1) is acquired, the processor 23 groups KF(1), KF(2), and KF(3) and associates them with a tracked object ID (1). Thereafter, when frame (k) is acquired, the processor 23 similarly executes grouping, and KF(1) and KF(2) are grouped into a set (1), and KF(4) and KF(5) are grouped into another set (2). Here, KF(3) satisfies the termination condition and is excluded from the grouping.

[0065] The processor 23 determines the similarity of the set (1) and the set (2) with the set at the time when the frame (k-1) was acquired. The determination of the similarity is performed, for example, by calculating the Simpson coefficient, but is not limited to this determination method. As another example, the Jaccard coefficient or the Dice coefficient may be used. The larger the Simpson coefficient, the more similar the two sets are. In the example of FIG. 12, since the set (1) including KF(1) and KF(2) has a higher similarity, the set (1) is associated with the tracked object ID (1). In other words, the set (1) inherits the tracked object ID (1). The set (2) is associated with another tracked object ID (2).

[0066] For example, suppose that when frame (k+1) is acquired, KF(2) is classified into set (2) instead of set (1). In this case, set (1) containing only KF(1) has a higher similarity to set (1) when frame (k) is acquired, and so it inherits the tracked object ID (1).

[0067] In this way, the processor 23 manages the identifiers according to the similarity of the groups at different times, and this management allows the control of tracking of the same object to continue appropriately.

[0068] As described above, the object tracking device 20 according to the present embodiment allows overlapping of detection results in the process of tracking multiple detection targets by the above configuration. Therefore, the object tracking device 20 can track multiple objects with high accuracy without causing a chain of misassociations.

[0069] Although the embodiments according to the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions included in each component or each step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined into one or divided. Although the embodiments according to the present disclosure have been described mainly with respect to the device, the embodiments according to the present disclosure can also be realized as a method including steps executed by each component of the device. The embodiments according to the present disclosure can also be realized as a method executed by a processor included in the device, a program, or a storage medium on which a program is recorded. It should be understood that these are also included in the scope of the present disclosure.

[0070] In the above embodiment, the observation value, which is the position of the detection target, is the position of the detection target, which is the detection result of the sensor data from the imaging device 10. Here, the object tracking system 1 may be configured to perform detection in parallel using a millimeter wave sensor, a detection device for reflected waves of laser light, etc. in addition to the imaging device 10. In this configuration, the object tracking system 1 can track multiple objects with higher accuracy by associating observation values ​​determined to be the same detection target with each other. Hereinafter, "fusion" refers to associating observation values ​​determined to be the same object from multiple observation values ​​obtained by physically different sensing methods with each other, taking into account their respective errors. In other words, fusion is a process that allows multiple observation values ​​obtained by different sensing methods to be linked to one detection target in a redundant manner. Since the new observation value generated by fusion is based on the detection results of multiple sensor data, the accuracy of the position of the detection target can be improved. In addition, the processor 23 does not discard the observation value that has not been fused, so that the complementarity is maintained. The process related to fusion may be executed as data processing (preprocessing) prior to direct object tracking.

[0071] When updating an observation value by fusion, the processor 23 applies the above-mentioned data association algorithm that allows overlapping as it is. The processor 23 selects one of the observation values ​​to be fused, with the error ellipse of the one observation value to be fused as the upper limit range, with the smallest Mahalanobis distance as the other observation value to be fused. FIG. 13 is a diagram for explaining fusion. In the example of FIG. 13, the observation value list A is an observation value obtained as a detection result of the sensor data of the imaging device 10, for example. Moreover, the observation value list B is an observation value obtained as a detection result of the sensor data of the millimeter wave sensor, for example. The processor 23 executes fusion and integrates the observation value list A and the observation value list B into the observation value list AB. For example, a1 in the observation value list A is fused with b1 in the observation value list B to obtain an observation value a1b1 with high accuracy. Moreover, an observation value that has not been fused, such as a4, is left as it is in the observation value list AB.

[0072] The processor 23 may perform fusion in a superimposed manner. Since the error of the fused observation value is necessarily small, it is possible to obtain an observation value with higher accuracy and precision. FIG. 14 is a diagram for explaining the superimposed application of fusion. In the example of FIG. 13, the observation value list AB is the same as that of FIG. 13. Moreover, the observation value list C is an observation value obtained as a detection result of sensor data of a detection device for detecting a reflected wave of a laser beam, for example. The processor 23 performs fusion in a superimposed manner to integrate the observation value list A, the observation value list B, and the observation value list C into the observation value list ABC. The observation value list ABC has an observation value with higher accuracy, for example, a1b1c1.

[0073] Here, the fused observations can be treated in the same way as the non-fused observations. In other words, data association is performed in the same way for both the fused and non-fused observations. Therefore, even if fusion is performed, the algorithm after data association is the same as in the above embodiment.

[0074] In the above embodiment, the object tracking system 1 includes the imaging device 10, the object tracking device 20, and the display 30, but at least two of these may be integrated. For example, the function of the object tracking device 20 may be mounted on the imaging device 10. In this case, the imaging device 10 may include the above-mentioned storage unit 22 and output interface 24 in addition to the imaging optical system 11, the imaging element 12, and the processor 13. Furthermore, the processor 13 may execute the processing performed by the processor 23 in the above embodiment on the moving image output by the imaging device 10. With such a configuration, the imaging device 10 that performs object tracking may be realized.

[0075] In the present disclosure, the term "mobile body" includes vehicles, ships, and aircraft. In the present disclosure, the term "vehicle" includes, but is not limited to, automobiles and industrial vehicles, and may include railroad vehicles, lifestyle vehicles, and fixed-wing aircraft that run on runways. Automobiles include, but are not limited to, passenger cars, trucks, buses, motorcycles, trolley buses, and the like, and may include other vehicles that run on roads. Industrial vehicles include industrial vehicles for agriculture and construction. Industrial vehicles include, but are not limited to, forklifts and golf carts. Industrial vehicles for agriculture include, but are not limited to, tractors, cultivators, transplanters, binders, combines, and lawnmowers. Industrial vehicles for construction include, but are not limited to, bulldozers, scrapers, excavators, cranes, dump trucks, and road rollers. Vehicles include those that run by human power. Here, the classification of vehicles is not limited to the above. For example, automobiles may include industrial vehicles that can run on roads, and the same vehicle may be included in multiple classifications. In the present disclosure, the vessel includes a marine jet, a boat, and a tanker. In the present disclosure, the aircraft includes a fixed-wing aircraft and a rotorcraft. [Explanation of symbols]

[0076] 1. Object Tracking System 10. Imaging device 11 Imaging optical system 12 Image sensor 13 Processors 20 Object Tracking Device 21 Input Interface 22 Memory section 23 Processors 24 Output Interfaces 30 Display 40 Object 40A Pedestrian 40B car 40C Bicycle 41 Image Space 42 Image of an object 43 Representative points 44 Reference plane 45 mass points 46 Virtual Space 100 vehicles

Claims

1. an input interface for acquiring sensor data; a processor that detects a plurality of detection targets from the sensor data and tracks each of the plurality of detection targets using a Kalman filter; an output interface for outputting a detection result of the detection object; The processor allows overlapping of detection results in a process of tracking the plurality of detection targets; The object tracking device, in which, when the multiple detected detection targets can be considered to be the same object, the processor represents the one with the smallest estimation range based on the probability density distribution of the positions of the multiple detection targets as the one that represents the same object.

2. The object tracking device according to claim 1 , wherein the processor controls the Kalman filter according to an initial state, a tracking preparation state, and a tracking state of the detection target.

3. The object tracking device according to claim 1 , wherein the processor sets the Kalman filter to a tracking state when the same detection target is detected consecutively.

4. The object tracking device according to claim 1 , wherein the processor stops tracking using the Kalman filter when the same detection target has not been detected a predetermined number of times in succession.

5. 5. The object tracking device according to claim 1, wherein the processor tracks the positions and velocities of mass points representing each of the plurality of detection targets using a virtual space, which is a two-dimensional space in which the value in the z-axis direction is a predetermined fixed value in a coordinate system consisting of three axes, i.e., an x-axis, a y-axis, and a z-axis in real space.

6. acquiring sensor data; detecting a plurality of detection targets from the sensor data, and tracking each of the plurality of detection targets using a Kalman filter; and outputting a detection result of the detection target. The tracking processor allows overlapping of detection results in the process of tracking the plurality of detection targets; The object tracking method, in which, when the detected multiple detection targets can be considered to be the same object, the processor represents the one with the smallest estimation range based on a probability density distribution of the positions of the multiple detection targets as the one that represents the same object.

7. an input interface for acquiring a plurality of sensor data obtained by different sensing methods; a processor that detects a plurality of detection targets from the plurality of sensor data and performs data processing to track each of the plurality of detection targets using a Kalman filter; The processor allows the detection results of the plurality of sensor data to be redundantly linked to one of the plurality of detection targets; The object tracking device, in which, when the multiple detected detection targets can be considered to be the same object, the processor represents the one with the smallest estimation range based on the probability density distribution of the positions of the multiple detection targets as the one that represents the same object.

8. Acquiring a plurality of sensor data obtained by different sensing methods; detecting a plurality of detection targets from the plurality of sensor data, and performing data processing for tracking each of the plurality of detection targets using a Kalman filter; the processor that performs the data processing allows detection results of the plurality of pieces of sensor data to be redundantly linked to one of the plurality of detection targets; The object tracking method, in which, when the detected multiple detection targets can be considered to be the same object, the processor represents the one with the smallest estimation range based on a probability density distribution of the positions of the multiple detection targets as the one that represents the same object.

Citation Information

Patent Citations

  • Rear side watching device

    JP1999321494A

  • Apparatus and method for detecting object

    JP2005311691A

  • Obstacle tracking system and method

    JP2007249309A