Object tracking device and object tracking method

Abstract

The present application provides an object tracking device and method capable of tracking multiple objects with high precision. The object tracking device (20) has an input interface (21) for obtaining sensor data, a processor (23) for detecting multiple detection objects from the sensor data and tracking the multiple detection objects using a Kalman filter, and an output interface (24) for outputting detection results of the detection objects, the processor (23) allowing repetition of the detection results in the process of tracking the multiple detection objects.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to Japanese Patent Application No. 2020-180783 (filed on October 28, 2020), and the entire contents of the application are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to an object tracking device and an object tracking method. BACKGROUND

[0004] A technique of detecting a surrounding object and tracking the detected object to predict a motion is known. For example, Patent Literature 1 discloses a device that processes an image signal output from a vehicle-mounted camera that acquires an image of a vehicle periphery, and detects whether or not a vehicle and a pedestrian that approach exist, and displays a mark of a quadrangular frame attached to the vehicle and the pedestrian that approach.

[0005] PRIOR ART DOCUMENTS

[0006] PATENT LITERATURE

[0007] Patent Literature 1: Japanese Patent Application Publication No. H11-321494 SUMMARY

[0008] An object tracking device according to an embodiment has:

[0009] an input interface that acquires sensor data;

[0010] a processor that detects a plurality of detection targets from the sensor data, and respectively tracks the plurality of detection targets using a Kalman filter; and

[0011] an output interface that outputs a detection result of the detection targets,

[0012] the processor allows repetition of the detection result in the tracking process of the plurality of detection targets.

[0013] An object tracking method according to an embodiment includes:

[0014] acquiring sensor data;

[0015] detecting a plurality of detection targets from the sensor data, and respectively tracking the plurality of detection targets using a Kalman filter; and

[0016] outputting a detection result of the detection targets,

[0017] in the tracking, allowing repetition of the detection result in the tracking process of the plurality of detection targets.

[0018] An object tracking apparatus of an embodiment has:

[0019] an input interface that acquires a plurality of sensor data obtained by different sensing methods; and

[0020] a processor that performs data processing for detecting a plurality of detection objects from the plurality of sensor data and tracking the plurality of detection objects respectively using Kalman filters,

[0021] the processor allows repeatedly associating detection results of the plurality of sensor data with one of the plurality of detection objects.

[0022] An object tracking method of an embodiment includes:

[0023] acquiring a plurality of sensor data obtained by different sensing methods; and

[0024] performing data processing for detecting a plurality of detection objects from the plurality of sensor data and tracking the plurality of detection objects respectively using Kalman filters,

[0025] the performing of the data processing allows repeatedly associating detection results of the plurality of sensor data with one of the plurality of detection objects. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a block diagram that shows an outline structure of an object tracking system including an object tracking apparatus of an embodiment.

[0027] Figure 2 is a diagram that shows a vehicle and a detection object of an object tracking system in which Figure 1 is mounted.

[0028] Figure 3 is a flowchart that shows an example of processing of tracking an image of an object on a dynamic image.

[0029] Figure 4 is a diagram that shows an example of an image of an object on a dynamic image.

[0030] Figure 5 is a diagram that explains a relationship of an object in a real space, an image of an object in a dynamic image, and a point in a virtual space.

[0031] Figure 6 is a diagram that shows an example of movement of a point in a virtual space.

[0032] Figure 7 is a diagram for explaining an action mode of a Kalman filter.

[0033] Figure 8 is a diagram for explaining data association.

[0034] Figure 9 is a diagram for explaining a Kalman filter.

[0035] Figure 10 is a diagram illustrating a layer structure of a tracking object ID management.

[0036] Figure 11 is a diagram for explaining determination of the same object.

[0037] Figure 12 is a diagram for explaining inheritance of a tracking object ID.

[0038] Figure 13 is a diagram for explaining fusion.

[0039] Figure 14 is a diagram for explaining repeated application of fusion. DETAILED DESCRIPTION

[0040] Hereinafter, an embodiment of the present application will be described with reference to the accompanying drawings. The drawings used in the following description are schematic. The dimensional ratios and the like on the drawings are not necessarily consistent with actual ones.

[0041] Figure 1 is a block diagram showing a schematic configuration of an object tracking system 1. An object tracking device 20 of an embodiment of the present application is included in the object tracking system 1. In the present embodiment, the object tracking system 1 includes a photographing device 10, the object tracking device 20, and a display 30. In addition, the object tracking system 1 is mounted on a vehicle 100, which is an example of a mobile body, as illustrated. Figure 2

[0042] The object tracking device 20 of the present embodiment acquires a dynamic image from the photographing device 10 as sensor data. That is, in the present embodiment, the sensor for detecting a plurality of detection targets is the photographing element 12 of the photographing device 10, which photographs visible light. However, the object tracking system 1 is not limited to one in which the photographing device 10 is used as the sensor. For example, the object tracking system 1 can include a sensor for detecting infrared light, and the sensor can be used as the sensor for detecting a plurality of detection targets. Figure 1 ​The object tracking system 1 can have a different device from the imaging device 10 as long as it is a device that detects a plurality of detection objects. As another example, the object tracking system 1 can have a measurement device that measures a distance to a detection object based on a reflected wave of irradiated laser light instead of the imaging device 10. As another example, the object tracking system 1 can have a detection device that has a millimeter wave sensor instead of the imaging device 10. In addition, as another example, the object tracking system 1 can have the imaging device 10 that has the imaging element 12 that images light other than visible light. In addition, the object tracking system 1 can have at least one of the imaging device 10 that takes visible light as an object, the measurement device that measures a distance to a detection object based on a reflected wave of irradiated laser light, the detection device that has a millimeter wave sensor, and the imaging device 10 that takes light other than visible light as an object.

[0043] In addition, in the present embodiment, the object tracking system 1 is mounted on a mobile body, and takes an object 40 (refer to Figure 2 ) around the moving mobile body as a detection object. However, the object tracking system 1 is not limited to the structure mounted on a mobile body. As another example, the object tracking system 1 can be used in a facility such as a factory, and takes a work person, a transport robot, a manufactured product, and the like as a detection object. In addition, as another example, the object tracking system 1 can be used in a welfare facility for the elderly and the like, and takes an indoor elderly person, a work person, and the like as a detection object. In addition, the object tracking system 1 can track an object not only for the safety of travel or movement, but also for the purpose of improving work efficiency, quality management, or improving productivity in, for example, an agricultural and industrial site. Here, in the present application, the object that is a detection object of the object tracking device 20 includes not only a mobile body and the like, but also a person.

[0044] As illustrated in Figure 2 , in the present embodiment, the x-axis direction in the coordinates of the actual space is the width direction of the vehicle 100 in which the imaging device 10 is provided. The y-axis direction is the backward direction of the vehicle 100. The x-axis direction and the y-axis direction are directions parallel to the road surface on which the vehicle 100 is present. The z-axis direction is a direction perpendicular to the road surface. The z-axis direction can be referred to as a plumb direction. The x-axis direction, the y-axis direction, and the z-axis direction are orthogonal to each other. The x-axis direction, the y-axis direction, and the z-axis direction are not limited to this. The x-axis direction, the y-axis direction, and the z-axis direction can be replaced with each other.

[0045] The imaging device 10 includes an imaging optical system 11, an imaging element 12, and a processor 13.

[0046] The imaging device 10 can be provided at each position of the vehicle 100. The imaging device 10 includes a front camera, a left camera, a right camera, and a rear camera, but is not limited to these. The front camera, the left camera, the right camera, and the rear camera are provided at the vehicle 100 in a manner that enables imaging of the periphery of the front, the left, the right, and the rear of the vehicle 100, respectively. In the embodiment described below as an example, as shown in FIG. 1, the imaging device 10 is mounted on the vehicle 100 in a manner that the optical axis direction is directed downward more than the horizontal direction to enable imaging of the rear of the vehicle 100. Figure 2

[0047] The imaging optical system 11 can include one or more lenses. The imaging element 12 can include a charge-coupled device image sensor (CCD) or a complementary MOS image sensor (CMOS).

[0048] The imaging element 12 converts an image (subject image) of an object imaged on the imaging surface of the imaging element 12 by the imaging optical system 11 into an electric signal. The imaging element 12 can image a moving image at a prescribed frame rate. A frame is each still image constituting a moving image. The number of images that can be imaged in one second is referred to as the frame rate. The frame rate can be, for example, 60 fps (frames per second) or 30 fps.

[0049] The processor 13 controls the entire imaging device 10 and performs various image processing on the moving image output from the imaging element 12. The image processing performed by the processor 13 can include any of distortion correction, brightness adjustment, contrast adjustment, gamma correction, and the like.

[0050] The processor 13 can be constituted by one or a plurality of processors. The processor 13 includes, for example, one or a plurality of circuits or units configured to execute one or a plurality of data computation processes or processing by executing instructions stored in an associated memory. The processor 13 includes one or a plurality of processors, microprocessors, microcontrollers, application specific integrated circuits (ASICs), digital signal processing devices (DSPs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), or any combination of these devices or configurations or other known devices or configurations.

[0051] ​The object tracking device 20 includes an input interface 21, a storage unit 22, a processor 23, and an output interface 24.

[0052] The input interface 21 can be configured to communicate with the imaging device 10 via a wired or wireless communication component. The input interface 21 acquires moving images from the imaging device 10 as sensor data. The input interface 21 can correspond to the transmission method of the image signal sent by the imaging device 10. The input interface 21 can also be referred to as an input unit or an acquisition unit. The imaging device 10 and the input interface 21 can be connected via an in-vehicle communication network such as CAN (control area network).

[0053] Storage unit 22 is a storage device for data and programs required for processing by storage processor 23. For example, storage unit 22 temporarily stores moving images acquired from imaging device 10. For example, storage unit 22 stores data generated by processing by processor 23. Storage unit 22 can be constructed using any one or more of the following: semiconductor memory, magnetic memory, and optical memory. Semiconductor memory can include volatile memory and non-volatile memory. Magnetic memory can include, for example, hard disk and magnetic tape. Optical memory can include, for example, CD (compact disc), DVD (digital versatile disc), and BD (blu-ray disc).

[0054] Processor 23 controls the entire object tracking device 20. Processor 23 identifies the images of objects included in the dynamic images acquired through input interface 21. Processor 23 maps and transforms the coordinates of the identified object images into virtual space 46 (see reference). Figure 6 The coordinates of object 40 are tracked, and the particle 45 representing object 40 in virtual space 46 (refer to) is tracked. Figure 5 The position and velocity of the particle 45. The particle 45 is a point with mass but no size. The virtual space 46 is a two-dimensional space in the coordinate system formed by the x, y, and z axes of the actual space, with the value of the z-axis direction set to a fixed value. The processor 23 can also map and transform the coordinates of the tracked particle 45 in the virtual space 46 to coordinates on the dynamic image.

[0055] Further, the processor 23 detects a plurality of detection objects from the dynamic image and tracks the plurality of detection objects using a Kalman filter respectively. In a case where a plurality of detection objects are detected, if their images overlap in the dynamic image, a tracking error or a decrease in accuracy can occur in the related art. In the present embodiment, the processor 23 establishes a correspondence between one or more Kalman filters and the plurality of detection objects respectively, and thus can avoid such a problem. Further, the processor 23 manages an observation value, a Kalman filter, and intrinsic identification information of a tracked object (hereinafter, “tracked object ID”) in each layer (hierarchy). The processor 23 determines whether or not the tracked objects are the same object and performs a process of establishing a correspondence between the observation value, the Kalman filter, and the tracked object ID. Thereby, the accuracy of tracking of the plurality of detection objects can be further improved. Details of the process performed by the processor 23 will be described later. The processor 23 can include a plurality of processors as with the processor 13 of the imaging device 10. Further, the processor 23 can be configured by a combination of a plurality of devices as with the processor 13.

[0056] The output interface 24 is configured to output an output signal from the object tracking device 20. The output interface 24 can also be referred to as an output section. The output interface 24 can output, for example, a detection result of a detection object such as a coordinate of the particle 45.

[0057] The output interface 24 can include a physical connector and a wireless communication device. The output interface 24 can be connected to a network of the vehicle 100 such as a CAN, for example. The output interface 24 can be connected to the display 30, a control device of the vehicle 100, an alarm device, and the like through a communication network such as a CAN. Information output from the output interface 24 can be appropriately used in each of the display 30, the control device, and the alarm device.

[0058] The display 30 can display a dynamic image output from the object tracking device 20. The display 30 can have a function of accepting a coordinate of the particle 45 indicating a position of an image of an object from the object tracking device 20, generating an image element (for example, a warning displayed together with an approaching object) based on this, and superimposing it in a dynamic image. The display 30 can employ various types of devices. For example, the display 30 can employ a liquid crystal display (LCD), an organic EL (electro-luminescence) display, an inorganic EL display, a plasma display (PDP), a field emission display (FED), an electrophoretic display, a twist ball display, or the like.

[0059] Hereinafter, the operation of the object tracking device 20 will be described with reference to the flowchart of FIG. 8. Figure 3The flowchart below illustrates the details of the object tracking method performed by the object tracking device 20. The object tracking device 20 can be configured to read a program recorded in a non-transitory computer-readable medium and execute the processing described below by the processor 23. Non-transitory computer-readable media include, but are not limited to, magnetic storage media, optical storage media, opto-magnetic storage media, and semiconductor storage media. Magnetic storage media include magnetic disks, hard disks, and magnetic tapes. Optical storage media include optical discs such as CDs, DVDs, and Blu-ray discs. Semiconductor storage media include ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), and flash memory.

[0060] Figure 3 The flowchart illustrates the process of acquiring consecutive frames of a moving image and executing them by processor 23. According to... Figure 3 The flowchart shows that the processor 23 of the object tracking device 20 tracks (traces) the image 42 of the object each time a frame of the dynamic image is acquired. Figure 4 The position of ). For example Figure 2 As shown, there are multiple objects 40 that are detected, including pedestrians 40A, cars 40B, and bicycles 40C. Furthermore, the objects 40 are not limited to moving objects and people, but can include various objects such as obstacles on the road. In the following description of the object tracking method, one of the multiple objects 40 (specifically, pedestrian 40A) included in the dynamic image of the camera device 10 installed behind the vehicle 100 will be used. Other objects 40 (e.g., cars 40B and bicycles 40C) are also tracked using the same process.

[0061] The processor 23 acquires frames of the dynamic image from the shooting device 10 through the input interface 21 (step S101). Figure 4 An example of a single frame of a moving image is shown. Figure 4 In the example, the image of the object 40 behind the vehicle 100 (image 42) is displayed in a two-dimensional image space 41 composed of the u and v coordinate systems. The u coordinate is the horizontal coordinate of the image. The v coordinate is the vertical coordinate of the image. In 4, the origin of the u and v coordinates is the point at the upper left end of the image space 41. In addition, the u coordinate is positive from left to right. The v coordinate is positive from top to bottom.

[0062] Processor 23 identifies object images 42 from each frame of the dynamic image through image recognition (step S102). The methods for identifying object images 42 include various known methods. For example, methods for identifying object images 42 include shape recognition based on objects such as cars and people, template matching-based methods, and methods that calculate feature quantities from the image and use them for matching. Feature quantities can be calculated using a function approximator that can learn the relationship between input and output. For example, a neural network can be used as a function approximator that can learn the relationship between input and output.

[0063] Processor 23 maps and transforms the coordinates (u, v) of the image 42 of the object in image space 41 into virtual space 46 (refer to...). Figure 6 The coordinates (x′, y′) of the object are obtained (step S103). Normally, the coordinates (u, v) of the image space 41, which are two-dimensional coordinates, cannot be converted to coordinates (x, y, z) in the actual space. However, by specifying the height in the actual space and fixing the z-coordinate to a predetermined value, it is possible to map the coordinates (u, v) of the image space 41 to the coordinates (x′, y′) of a two-dimensional virtual space 46 corresponding to the coordinates (x, y, z0) in the actual space (z0 is a fixed value). Here, in this embodiment, the virtual space 46 is set to two dimensions, but it can also be set to three dimensions depending on the input information (the type of sensor).

[0064] like Figure 4 As shown, a representative point 43 is specifically located at the center of the lowest part of the image 42 of the object. For example, the representative point 43 can be set as the lowest position of the v-coordinate and the center position of the range of the u-coordinate of the area occupied by the image 42 of the object in the image space 41. Assume that the representative point 43 is the position where the road surface or ground of the object 40 corresponding to the image 42 of the object is in contact.

[0065] exist Figure 5The diagram illustrates the relationship between an object 40 located in three-dimensional real space and the image 42 of an object in two-dimensional image space 41. When the internal parameters of the imaging device 10 are known, the direction (x, y, z) from the center of the imaging optical system 11 of the imaging device 10 toward the corresponding coordinates (x, y, z) in the real space is calculated based on the coordinates (u, v) in the image space 41. The internal parameters of the imaging device 10 include information such as the focal distance of the imaging optical system 11, distortion, and the pixel size of the imaging element 12. In the real space, the point where the straight line pointing toward the direction corresponding to the representative point 43 in the image space intersects with the reference plane 44 at z = 0 is designated as the mass point 45 of the object 40. The reference plane 44 corresponds to the road surface or ground where the vehicle 100 is located. The mass point 45 has three-dimensional coordinates (x, y, 0). Therefore, when the two-dimensional space at z = 0 is designated as the virtual space 46, the coordinates of the mass point 45 can be represented by (x′, y′). The coordinates (x′, y′) of particle 45 in virtual space 46 are equivalent to the coordinates (x, y) of a specific point of object 40 in the xy plane (z=0) when viewed from along the z-axis in actual space. The specific point is the point corresponding to particle 45.

[0066] like Figure 6 As shown, processor 23 tracks the position (x′, y′) and velocity (vx′, vy′) of a particle 45 in virtual space 46, which is mapped from the representative point 43 of the object's image 42 to the virtual space 46 (step S104). Using the information of the position (x′, y′) and velocity (vx′, vy′) of the particle 45, processor 23 can predict the range of the position (x′, y′) of the particle 45 in consecutive frames. Processor 23 can identify the particle 45 in the next frame that falls within the predicted range as the particle 45 corresponding to the image 42 of the tracked object. Processor 23 updates the position (x′, y′) and velocity (vx′, vy′) of the particle 45 sequentially each time a new frame is received.

[0067] Tracking of the particle 45 can be achieved, for example, by using an estimation using a Kalman filter based on a state-space model. By performing prediction / estimation using a Kalman filter, robustness against undetectable and false detections of the object 40 being detected is improved. Typically, it is difficult to describe the image 42 of an object in image space 41 using an appropriate model to describe motion. Therefore, it is difficult to simply perform high-precision position estimation of the image 42 of an object in image space 41. For the object tracking device 20 of the present invention, by mapping and transforming the image 42 of the object into a particle 45 in real space, a model describing motion in real space can be applied, thus improving the tracking accuracy of the image 42 of the object. In addition, by treating the object 40 as a particle 45 without size, tracking can be simple and easy.

[0068] Each time the processor 23 estimates a new position for the particle 45, in order to display the estimated position, it can map and transform the coordinates of the particle 45 in the virtual space 46 to the coordinates (u, v) in the image space 41 (step S105). The particle 45 located at coordinates (x′, y′) in the virtual space 46 can be mapped and transformed into the image space 41 as a point located at coordinates (x, y, 0) in the actual space. The coordinates (x, y, 0) in the actual space can be mapped to coordinates (u, v) in the image space 41 of the imaging device 10 using a known method. The processor 23 can convert between the coordinates (u, v) in the image space 41, the coordinates (x′, y′) in the virtual space 46, and the coordinates (x, y, 0) in the actual space.

[0069] In this embodiment, the processor 23 detects multiple objects from a dynamic image and tracks them individually. For example, in an image... Figure 2 In that scenario, processor 23 tracks pedestrian 40A, car 40B, and bicycle 40C respectively. Processor 23 uses virtual space 46 to track the position and velocity of particles 45 representing multiple detected objects. Processor 23 needs to prepare Kalman filters for each of the multiple objects 40 to execute... Figure 3 The object tracking method shown is as follows. In this embodiment, when the processor 23 detects the image 42 of a new object in the dynamic image, a new Kalman filter is generated, and tracking is performed when the start-up condition is met. Thus, more than one Kalman filter is prepared for each of the multiple objects 40 that are being detected. However, if Kalman filters are continuously generated for transient new objects 40 (e.g., oncoming vehicles that will no longer be included in the dynamic image after a predetermined time), the number of Kalman filters increases beyond necessary limits, increasing the computational load and potentially causing processing delays in object tracking. In this embodiment, the processor 23 initializes the Kalman filters when the termination condition is met, thus avoiding processing delays.

[0070] Figure 7 This diagram illustrates the operating modes of the Kalman filter. The processor 23 controls the Kalman filter based on different states of the detected object: the initial state, the tracking preparation state, and the tracking state.

[0071] The initial state of the detected object is determined by the processor 23 based on the state of the image 42, which represents a new object in the dynamic image. At this time, the Kalman filter that establishes a correspondence with the detected object operates in "Mode 0". The Kalman filter in Mode 0 may also lack initial values ​​(position and velocity information). When the Kalman filter that establishes a correspondence with the detected object is in Mode 0, the processor 23 does not track the position of the detected object, i.e., it does not predict the range of the position (x′, y′) of the particle 45 in the next frame.

[0072] The tracking preparation state is when the image 42 of the newly identified object in the previous frame is also identified in the current frame. At this time, the Kalman filter that establishes a correspondence with the detected object operates in "Mode 1". The Mode 1 Kalman filter acquires the position (x′, y′) of the detected object's mass point 45, but because there is no information about the detected object's position from the previous frame, it does not acquire the velocity (v). x′ v y′ The Kalman filter in Mode 1 only has a subset of the necessary initial values ​​(position and velocity information). When the Kalman filter corresponding to the detected object is in Mode 1, the processor 23 does not track the position of the detected object.

[0073] When the Kalman filter is in mode 1, it performs processing such as confirming that image 42 of the object is not a false detection. For example... Figure 7 As shown, when the first termination condition is met, i.e., when it is determined to be a false detection or the image 42 of the object disappears, the Kalman filter's operating mode is initialized and returns to mode 0. This avoids starting tracking due to scattered false detections.

[0074] In addition, such as Figure 7 As shown, when the activation conditions are met, that is, when the Kalman filter is in mode 1 and the image 42 of the newly identified object in the previous two frames is also identified in the current frame, the Kalman filter operates in mode 2.

[0075] The tracking state is the state where the second termination condition is not met after the above-mentioned start condition is satisfied. The second termination condition is that the image 42 of the object disappears in a consecutive and predetermined number of frames up to the current frame. At this time, the Kalman filter that establishes a correspondence with the detected object operates in "Mode 2". The Kalman filter in Mode 2 has the necessary initial values ​​(position and velocity information) and can be immediately used for tracking control. When the Kalman filter that establishes a correspondence with the detected object is in Mode 2, the processor 23 tracks the position of the detected object.

[0076] like Figure 7 As shown, when the Kalman filter that establishes a correspondence with the detected object is in mode 2 and the second termination condition is met, the operation mode of the Kalman filter is initialized and returns to mode 0.

[0077] As described above, when the processor 23 continuously detects the same target, it sets the Kalman filter to tracking state (mode 2). Here, in this embodiment, the number of continuous detections is 2, but it can also be 3 or more. When the number of continuous detections is 3 or more, for example, the mode 1 state (tracking preparation state) can be maintained for a long time.

[0078] In addition, the processor 23 stops tracking using the Kalman filter in a case where the same detection target cannot be continuously detected in a prescribed number. Here, the prescribed number is 5 in the present embodiment, but is not limited thereto. In tracking of the object using the Kalman filter, even if information of the position of the detection target acquired from the dynamic image is not available, it is possible to continue to predict the range of the position of the detection target. However, the error of the predicted range of the position increases as the number of frames in which such information is not available increases. The above-described prescribed number can be determined in accordance with the magnitude of the error.

[0079] The processor 23 can perform system control of a plurality of Kalman filters for different states by setting the above-described operation mode to the Kalman filter and dynamically changing the setting.

[0080] Figure 8 is a diagram for explaining data association. The data association is the correspondence relationship of a plurality of observation values and a plurality of Kalman filters. Here, the observation value is the position of the detection target. The processor 23 gives an identification to distinguish the plurality of observation values and the plurality of Kalman filters. In the present embodiment, the processor 23 sets the plurality of observation values to observation value (1), observation value (2), observation value (3), …, respectively, for example, using serial numbers. In addition, the processor 23 sets the plurality of Kalman filters to KF (1), KF (2), KF (3), …, respectively, for example, using symbols and serial numbers.

[0081] In the present embodiment, the processor 23 associates data of M observation values and N Kalman filters. M is an integer of 2 or more. N is an integer of M or more. In the present embodiment, M is 3 and N is 5. Figure 8 In the example of, the processor 23 associates data of 3 observation values and 5 Kalman filters. The observation value (1) is the position of the pedestrian 40A detected in the frame (k) of the dynamic image. The observation value (2) is the position of the car 40B detected in the frame (k) of the dynamic image. The observation value (3) is the position of the bicycle 40C detected in the frame (k) of the dynamic image. In addition, the frame (k-1) is the frame immediately before the frame (k) in the dynamic image. The frame (k-2) is the frame 2 frames before the frame (k) in the dynamic image. The current frame is the frame (k).

[0082] Here, KF(2) is initialized for tracking of the pedestrian 40A until frame (k-1) because the second termination condition is satisfied. That is, the action mode of KF(2) is mode 0, and tracking of the position of the detected object is not performed. In addition, KF(5) is a Kalman filter newly prepared because a new bicycle 40C is recognized in frame (k-2). KF(5) is in mode 1 at frame (k-1), but becomes mode 2 because the activation condition is satisfied. The other Kalman filters continue tracking of the detected objects as mode 2 respectively from frame (k-2).

[0083] In Figure 8 In the example, the processor 23 associates KF(1) with the observation (1). The processor 23 associates KF(3) and KF(4) with the observation (2). In addition, the processor 23 associates KF(5) with the observation (3). As in the example of the observation (2), the processor 23 allows repetition of the detection results in the tracking processes of the plurality of detected objects. That is, the processor 23 performs prediction of the range of the position of the observation (2), the car 40B, using KF(3) and KF(4). As described above, local optimization can be performed by allowing repetition in data association. For example, in a method in which a plurality of observations are associated with a plurality of Kalman filters one-to-one (as an example, Hungarian method) without allowing repetition, a chain reaction of one false association can occur due to global optimization. In the present embodiment, since repetition is allowed, a problem of a chain reaction of false association does not occur. In addition, in the tracking processes, one or more Kalman filters are associated with one observation, and it is difficult to cause failure of tracking for whichever observation, and thus robustness can be improved.

[0084] As an example of associating a plurality of Kalman filters with one observation, for example, a case in which one object is recognized as two objects due to the influence of reflection of light or the like, and a new Kalman filter is associated with the other is considered. As described above, the plurality of Kalman filters associated with the detection are used, and control of tracking of the detected object is performed in parallel. However, for example, in a case in which the predicted position of the detected object is used to prevent collision of the vehicle 100, it is sometimes preferable to output one detection result having the highest confidence from the output interface 24. The processor 23 can determine a Kalman filter (hereinafter, "representative Kalman filter") indicating the detection result having the highest confidence from the error ellipses of the Kalman filters.

[0085] Figure 9 is a diagram for explaining the representative Kalman filter. In Figure 9In the example of FIG. 6, three Kalman filters, KF(p), KF(q), and KF(r), are associated with one observation. The processor 23 calculates error ellipses for the three Kalman filters, respectively. The error ellipses indicate an estimated range of a probability density distribution based on positions, and indicate that a position is located inside the ellipses with a prescribed probability (for example, 99%). The error ellipses are calculated using a standard deviation in the x' direction and a standard deviation in the y' direction, and the like. The processor 23 determines the Kalman filter having the smallest error ellipse as the representative Kalman filter. In the example of FIG. 6, KF(q) is the representative Kalman filter. Figure 9 In the example of FIG. 6, KF(r) is the representative Kalman filter.

[0086] As described above, in a case where the plurality of detected objects are considered to be the same object, the processor 23 can represent the object with the detected object having the smallest estimated range of the estimated range of the probability density distribution based on positions of the plurality of detected objects. Thus, the object tracking device 20 is also applicable to driving assistance such as collision prevention of the vehicle 100.

[0087] Here, as described above, the plurality of Kalman filters can be associated with one observation, but the plurality of observations can be associated with one object as the detected object. For example, in a case where the detected object is the automobile 40B and the automobile 40B temporarily disappears from the dynamic image due to a lane change or the like and then appears in the dynamic image again, a new observation can be associated as another object. It is preferable that the object tracking device 20 recognize each tracked object to manage the association with the observation to perform accurate tracking of the object. In the present embodiment, the processor 23 performs hierarchical management as described below, and performs grouping of the plurality of Kalman filters to determine whether or not to be associated with the same object.

[0088] Figure 10 is a diagram indicating a hierarchical structure of tracking object ID management in the present embodiment. As shown in Figure 10 , the processor 23 manages the observation, the Kalman filter, and the tracking object ID in each layer. In addition, the processor 23 can perform accurate tracking of the object by associating the observation, the Kalman filter, and the tracking object ID. Here, the tracking object ID is unique identification information of the tracked object as described above. If the tracking object IDs associated with the plurality of observations or the plurality of Kalman filters are the same, the observations or the Kalman filters are associated with tracking of the same object.

[0089] As described above, the processor 23 generates the Kalman filter for the new observation, and associates one or more Kalman filters with one observation. The processor 23 also associates the Kalman filter with the tracking object ID. Figure 11is a diagram for explaining the determination of the same object. The processor 23 groups the plurality of Kalman filters by clustering such as DBSCAN (density-based spatial clustering of applications with noise) or the like. The processor 23 determines that the Kalman filters belong to one group in a case where the centers of the error ellipses of the plurality of Kalman filters are included in a prescribed range. In Figure 11 In the example of FIG. 8, the prescribed range is represented by a circle. Further, KF(p), KF(q), and KF(r) are one group. Here, the prescribed range can vary depending on the size of the tracked object. For example, if the tracked object is the automobile 40B, the prescribed range can be set to be larger than in a case where the tracked object is the pedestrian 40A. Further, the prescribed range can be fixed regardless of the kind of the tracked object. The method of clustering is not limited to DBSCAN. Clustering can also be performed by another method such as a k-means method or the like.

[0090] The processor 23 performs grouping of the plurality of Kalman filters when acquiring the frame of the dynamic image. Further, the processor 23 updates the correspondence of the observation value, the Kalman filter, and the tracked object ID. In Figure 10 In the example of FIG. 8, the processor 23 groups KF(1), KF(2), and KF(3), assigns "tracked object ID (1)" as an identifier to the object tracked using these Kalman filters, and performs tracking control of the object. Further, the processor 23 groups KF(4) and KF(5), assigns "tracked object ID (2)" as an identifier to the object tracked using these Kalman filters, and performs tracking control of the object.

[0091] Here, KF(1) and KF(2) are in correspondence with the observation value (1), and KF(3) is in correspondence with the observation value (2). The processor 23 can recognize the observation value (1) and the observation value (2) as positions of the same object identified by the tracked object ID (1) by grouping. The processor 23 controls tracking in a hierarchical structure in which the Kalman filters corresponding to the same object determined to be the same are associated and the detection results of the detection targets corresponding to these Kalman filters are also associated, and thus can perform tracking with no error and with high precision. The processor 23 can obtain a detection result with high confidence by comparing or selecting the detection results using, for example, the plurality of Kalman filters associated. Further, the processor 23 continues tracking of the object identified by the tracked object ID (1) using the observation value (1), KF(1), and KF(2) even in a case where, for example, the observation value (2) is lost or KF(3) is initialized. That is, robustness can be improved.

[0092] Here, the processor 23 can determine the Kalman filter having the smallest error ellipse as the representative Kalman filter for the plurality of Kalman filters belonging to the same group, as with the above (refer to Figure 9 ) The processor 23 can determine the Kalman filter having the smallest estimation range of the probability density distribution based on the positions of the plurality of detection objects among the Kalman filters corresponding to the same object as the representative of the group.

[0093] Figure 12 is a diagram for explaining inheritance of the tracking object ID. As described above, the processor 23 performs grouping of the plurality of Kalman filters at the time of acquisition of the frame of the dynamic image. The processor 23 appropriately inherits the tracking object ID according to the similarity in the case where the constitution of the group is changed.

[0094] In the example of Figure 12 , the processor 23 groups KF(1), KF(2), and KF(3) at the time of acquisition of the frame (k-1), and establishes a correspondence relationship with the tracking object ID (1). Next, at the time of acquisition of the frame (k), the processor 23 similarly performs grouping, and KF(1) and KF(2) are grouped into a set (1), and KF(4) and KF(5) are grouped into another set (2). Here, KF(3) satisfies the termination condition, and is excluded from the grouping target.

[0095] The processor 23 determines the similarity to the set at the time of acquisition of the frame (k-1) for the set (1) and the set (2). The determination of the similarity is performed by, for example, calculating the Simpson coefficient, but is not limited to this determination method. As another example, the Jaccard coefficient or the Dice coefficient or the like can also be used. The larger the Simpson coefficient, the more similar the two sets. In the example of Figure 12 , since the set (1) including KF(1) and KF(2) has a higher similarity, the set (1) establishes a correspondence relationship with the tracking object ID (1). That is, the set (1) inherits the tracking object ID (1). Another tracking object ID (2) establishes a correspondence relationship with the set (2).

[0096] Here, for example, at the time of acquisition of the frame (k+1), KF(2) is divided into the set (2) instead of the set (1). At this time, the set (1) including only KF(1) has a higher similarity to the set (1) at the time of acquisition of the frame (k), and thus directly inherits the tracking object ID (1).

[0097] As described above, the processor 23 manages the identification by the similarity of the group at different times. By this management, it is possible to appropriately continue the control of the tracking of the same object.

[0098] As described above, the object tracking device 20 of the present embodiment allows repetition of the detection results in the tracking process of multiple detection objects by the above-described configuration. Therefore, the object tracking device 20 can track multiple objects with high accuracy without a chain reaction of erroneous association.

[0099] Embodiments of the present application are described in accordance with the drawings and examples, but it should be noted that various modifications or variations based on the present application are easy for those skilled in the art. Therefore, it should be noted that these modifications or variations are included in the scope of the present application. For example, the functions and the like included in each of the constituent parts or each of the steps and the like can be reconfigured in a logically non-contradictory manner, and a plurality of constituent parts or steps and the like can be combined or divided into one. The embodiments of the present application are described centering on a device, but the embodiments of the present application can also be realized as a method of steps performed by each of the constituent parts of the device. The embodiments of the present application can also be realized as a method executed by a processor possessed by the device, a program, or a storage medium in which the program is recorded. It should be understood that these are also included in the scope of the present application.

[0100] In the above-described embodiments, the detection result of the sensor data from the imaging device 10, that is, the position of the detection object, is directly used as the observation value of the position of the detection object. Here, the object tracking system 1 can also be a structure in which detection is performed in parallel by a millimeter wave sensor, a detection device of reflected waves of laser light, or the like in addition to the imaging device 10. In the case of such a structure, the object tracking system 1 can track multiple objects with higher accuracy by associating observation values determined to be the same detection object with each other. Hereinafter, “fusion” means that a plurality of observation values obtained using different physical sensing methods are determined to be the same object, and these observation values are associated while the respective errors are taken into account. In other words, fusion is a process of repeatedly associating a plurality of observation values with one detection object while allowing different sensing methods to be used. Since a new observation value generated by fusion is based on the detection results of a plurality of sensor data, the accuracy of the position of the detection object can be improved. In addition, since the processor 23 does not discard observation values that are not fused, the complementarity thereof is maintained. The process related to fusion can also be performed as a prior data process (pre-processing) of direct object tracking.

[0101] When the processor 23 updates the observation value by fusion, the algorithm of the above-described data association that allows repetition is directly applied. The processor 23 selects one observation value with the smallest Mahalanobis distance as another observation value to be fused, with the error ellipse of one observation value to be fused as an upper limit range. Figure 13 is a diagram for explaining fusion. In Figure 13In the example of FIG. 10, the observation list A is, for example, an observation obtained as a detection result of sensor data of the imaging device 10. In addition, the observation list B is, for example, an observation obtained as a detection result of sensor data of a millimeter wave sensor. The processor 23 performs fusion, thereby integrating the observation list A and the observation list B in the observation list AB. For example, an observation of a1b1 of high precision is obtained by fusion of a1 of the observation list A and b1 of the observation list B. In addition, an observation such as a4 that is not fused is left as it is in the observation list AB.

[0102] The processor 23 can perform fusion in overlapping. Since an error of a fused observation is necessarily small, an observation of higher accuracy and precision can be obtained. Figure 14 FIG. 11 is a diagram for explaining repeated application of fusion. In Figure 14 In the example of FIG. 10, the observation list AB is the same as Figure 13 In addition, the observation list C is, for example, an observation obtained as a detection result of sensor data of a detection device of a reflected wave of laser light. The processor 23 integrates the observation list A, the observation list B, and the observation list C in the observation list ABC by performing fusion in overlapping. The observation list ABC has, for example, an observation of a1b1c1 of higher precision.

[0103] Here, a fused observation can be processed in the same manner as an unfused observation. That is, both a fused observation and an unfused observation are data-associated in the same manner. Therefore, in a case where fusion is performed, an algorithm after data association is the same as in the above-described embodiment.

[0104] In the above-described embodiment, at least two of the imaging device 10, the object tracking device 20, and the display 30 of the object tracking system 1 can be an integrated structure. For example, a function of the object tracking device 20 can be mounted on the imaging device 10. At this time, the imaging device 10 can have the above-described storage section 22 and the output interface 24 in addition to the imaging optical system 11, the imaging element 12, and the processor 13. In addition, the processor 13 can perform processing performed by the processor 23 in the above-described embodiment on a moving image output from the imaging device 10. With this structure, the imaging device 10 that performs tracking of an object can also be realized.

[0105] The "moving body" in the present application includes a vehicle, a ship, and an airplane. The "vehicle" in the present application includes an automobile and an industrial vehicle, but is not limited thereto, and can include a railway vehicle, a living vehicle, and a fixed-wing aircraft that runs on a runway. The automobile includes a passenger car, a truck, a bus, a two-wheeled vehicle, and a trolleybus, but is not limited thereto, and can include other vehicles that run on a road. The industrial vehicle includes an industrial vehicle for agriculture and construction. The industrial vehicle includes a forklift and a golf cart, but is not limited thereto. The industrial vehicle for agriculture includes a tractor, a cultivator, a transplanter, a harvester, a combine harvester, and a mower, but is not limited thereto. The industrial vehicle for construction includes a bulldozer, a shovel, a digger, a crane, a self-dumping car, and a road roller, but is not limited thereto. The vehicle includes a vehicle that runs by human power. Here, the classification of the vehicle is not limited to the above. For example, the automobile can include an industrial vehicle that can run on a road, and can include the same vehicle in a plurality of classifications. The ship in the present application includes a motorboat, a small boat, and a tanker. The airplane in the present application includes a fixed-wing aircraft and a rotary-wing aircraft.

[0106] BRIEF DESCRIPTION OF DRAWINGS

[0107] 1: object tracking system;

[0108] 10: imaging device;

[0109] 11: imaging optical system;

[0110] 12: imaging element;

[0111] 13: processor;

[0112] 20: object tracking device;

[0113] 21: input interface;

[0114] 22: storage unit;

[0115] 23: processor;

[0116] 24: output interface;

[0117] 30: display;

[0118] 40: object;

[0119] 40A: pedestrian;

[0120] 40B: automobile;

[0121] 40C: bicycle;

[0122] 41: image space;

[0123] 42: image of object;

[0124] 43: representative point;

[0125] 44: reference surface;

[0126] 45: mass point;

[0127] 46: virtual space;

[0128] 100: vehicle.

Claims

1. An object tracking device, wherein, have: Input interface to acquire sensor data; The processor detects multiple objects based on the sensor data and tracks each of the multiple objects using a Kalman filter. as well as The output interface outputs the detection results of the detected object. The processor is also configured to allow the repetition of detection results during the tracking process of a plurality of the detected objects. In cases where multiple detected objects are considered to be the same object, the processor is further configured to represent the object as the detected object with the smallest estimated range among the estimated ranges of the probability density distributions of the positions of the multiple detected objects.

2. The object tracking device according to claim 1, wherein, The processor is also configured to control the Kalman filter for the initial state, tracking preparation state, and tracking state of the detected object, respectively.

3. The object tracking device according to claim 2, wherein, The processor is also configured to set the Kalman filter to a tracking state when the same detection object is detected continuously.

4. The object tracking device according to any one of claims 1 to 3, wherein, The processor is also configured to stop using the Kalman filter for tracking if the same object cannot be detected consecutively for a specified number of times.

5. The object tracking device according to any one of claims 1 to 3, wherein, The processor is also configured to use a virtual space to track the position and velocity of the mass points representing the plurality of the detected objects, wherein the virtual space is a two-dimensional space in which the value of the z-axis direction in a coordinate system composed of the three axes of the actual space (x-axis, y-axis, and z-axis) is set to a predetermined fixed value.

6. The object tracking device according to claim 4, wherein, The processor is also configured to use a virtual space to track the position and velocity of the mass points representing the plurality of the detected objects, wherein the virtual space is a two-dimensional space in which the value of the z-axis direction in a coordinate system composed of the three axes of the actual space (x-axis, y-axis, and z-axis) is set to a predetermined fixed value.

7. An object tracking method, wherein, include: Acquire sensor data; Multiple detection objects are detected based on the sensor data, and a Kalman filter is used to track each of the multiple detection objects. as well as Output the detection result of the object being detected. The processor performing the tracking allows for the repetition of detection results during the tracking process of multiple objects. In cases where multiple detected objects are considered to be the same object, the processor is further configured to represent the object as the detected object with the smallest estimated range among the estimated ranges of the probability density distributions of the positions of the multiple detected objects.

8. An object tracking device, wherein, have: The input interface acquires data from multiple sensors obtained through different sensing methods. as well as The processor performs data processing for detecting multiple objects based on data from multiple sensors, and for tracking each of the multiple detected objects using a Kalman filter. The processor is also configured to allow the detection results of multiple sensor data to be repeatedly associated with one of multiple detection objects. In cases where multiple detected objects are considered to be the same object, the processor is further configured to represent the object as the detected object with the smallest estimated range among the estimated ranges of the probability density distributions of the positions of the multiple detected objects.

9. An object tracking method, wherein, include: Acquire data from multiple sensors using different sensing methods; as well as Data processing is performed to detect multiple objects based on data from multiple sensors, and to track each of the multiple objects using a Kalman filter. The processor performing the data processing allows for the repeated association of detection results from multiple sensor data with one of multiple detection objects. In cases where multiple detected objects are considered to be the same object, the processor is further configured to represent the object as the detected object with the smallest estimated range among the estimated ranges of the probability density distributions of the positions of the multiple detected objects.

Images