Target tracking device and target tracking method

Abstract

A target tracking device comprising: a detection unit (11; 21; 31; 41) that detects features including the position of at least one target from sensor data; a tracking unit (12; 22; 32; 42) that tracks the target based on the detected features and outputs a tracking trail of the tracked target; and a track classification unit (13; 23; 33; 43) that determines which of a plurality of predetermined movement patterns the output tracking trail corresponds to.

Description

[Technical field]

[0001] The present disclosure relates to target tracking techniques. [Background technology]

[0002] There is a demand for monitoring targets for various purposes, such as preventive maintenance or guiding advertising. As a means for continuously monitoring a target, there is a technology for detecting a target using a non-contact sensor such as a camera, radar or laser, and tracking the detected target. Prior art target detection and tracking technologies include those disclosed in the following patent documents. Patent document 1 describes a method for detecting a target using a camera and a method for tracking the detected target. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 171498 Summary of the Invention [Problem to be solved by the invention]

[0004] According to the prior art, tracking of an observed target is performed based only on observation data relating to that target, and therefore there is a problem in that individual targets cannot be tracked with high accuracy in a multi-target crowded environment where multiple targets exist.

[0005] The present disclosure has been made to solve such problems, and aims to provide a target tracking technology that can track individual targets with high accuracy even in a crowded multi-target environment. [Means for solving the problem]

[0006] One aspect of a target tracking device according to an embodiment of the present disclosure includes a detection unit that detects features including a position of at least one target from sensor data, a tracking unit that tracks the target based on the detected features and outputs a tracking trail of the tracked target, and a trail classification unit that determines which of a plurality of predetermined movement patterns the output tracking trail corresponds to. Effect of the Invention

[0007] According to the target tracking technology according to the embodiment of the present disclosure, individual targets can be tracked with high accuracy even in a crowded environment with many targets. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of a target tracking device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a tracking unit according to the first embodiment. [Figure 3A] 4 is a flowchart illustrating a track classification learning process according to the first embodiment. [Figure 3B] 4 is a flowchart illustrating a track classification inference process according to the first embodiment. [Figure 4] FIG. 11 is a diagram illustrating an example of the configuration of a target tracking device according to a second embodiment. [Diagram 5] FIG. 11 is a diagram illustrating an example of the configuration of a target tracking device according to a third embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of the configuration of a target tracking device according to a fourth embodiment. [Figure 7] 2 is a diagram for explaining the operation of the target tracking device according to the first to fourth embodiments. FIG. [Figure 8] 11 is a diagram for explaining the operation of a target tracking device according to embodiment 3. FIG. [Figure 9] FIG. 2 is a diagram illustrating an example of a hardware configuration of a target tracking device according to the first to fourth embodiments. [Figure 10] FIG. 2 is a diagram illustrating an example of a hardware configuration of a target tracking device according to the first to fourth embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Various embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that components with the same or similar reference numerals in the drawings have the same or similar configurations or functions, and redundant descriptions of such components will be omitted.

[0010] Embodiment 1 <Configuration> A target tracking device according to a first embodiment of the present disclosure will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a diagram showing a configuration example of a target tracking device 10 according to the first embodiment, and Fig. 2 is a diagram showing a configuration example of a tracking unit 12 included in the target tracking device 10.

[0011] As shown in FIG. 1, one aspect of a target tracking device 10 according to the first embodiment includes a detection unit 11 that detects features including the position of at least one target from sensor data, a tracking unit 12 that tracks the target based on the detected features and outputs a tracking trail of the tracked target, and a track classification unit 13 that determines which of a plurality of predetermined movement patterns the output tracking trail corresponds to.

[0012] Another aspect of the target tracking device 10 includes a detection unit 11, a tracking unit 12, a track classification unit 13, and a track processing unit 14. The track processing unit 14 is an optional functional unit, and the target tracking device 10 does not need to include the track processing unit 14. As shown in FIG. 1, a target tracking system using the target tracking device 10 includes a sensor observation unit 1, the target tracking device 10 connected to the sensor observation unit 1, a memory unit 2 connected to the target tracking device 10, and a display unit 3 connected to the target tracking device 10.

[0013] (Sensor observation section) The sensor observation unit 1 is a functional unit that acquires sensor data related to a target obtained by sensing. In the present disclosure, the number of targets sensed by the sensor observation unit 1 is assumed to be multiple. However, this does not exclude the case where an embodiment using the technology of the present disclosure senses a single target.

[0014] The sensor observation unit 1 is, for example, a camera, a radar, or a laser sensor. For example, when the sensor observation unit 1 is a camera, the camera captures an image of a target to obtain a moving image consisting of multiple frames as sensor data. The sensor observation unit 1 outputs the obtained sensor data to the detection unit 11.

[0015] (Detection unit) The detection unit 11 detects a target from raw data such as an image for each frame output from the sensor observation unit 1, and outputs the feature amount of the detected target to the tracking unit 12. For example, in the case of target detection from a camera image, the detection unit 11 calculates the position and size of the target using a target detection algorithm such as SSD (Single Shot MultiBox Detector) or YOLO (You Look Only Once) for the image data output from the sensor observation unit 1. For example, if the target is a person, the detection unit 11 calculates the position and size of the person's head. In addition, the detection unit 11 may calculate the appearance feature amount of the target such as an RGB histogram, an HSV histogram, or a high-dimensional feature amount based on metrics-learning for the image data output from the sensor observation unit 1. The detection unit 11 outputs the feature amount of the target such as the position, size, appearance feature amount, etc. calculated for each frame to the tracking unit 12.

[0016] (Tracking part) The tracking unit 12 is a functional unit including a prediction unit 121 that predicts a predicted feature at a second time later than a first time from a first observation feature detected at a first time, a correlation unit 122 that determines a correlation between the predicted predicted feature and a second observation feature detected at the second time, and a filter unit 123 that performs filtering using the correlated second observation feature and the predicted predicted feature, and outputs time series data of the filtered feature as a tracking track.

[0017] The tracking unit 12 will be described in more detail. Based on the feature values ​​output from the detection unit 11, the tracking unit 12 determines the correlation between the predicted state of the target at the current time in the previous frame and the observed values ​​of the feature values ​​of the target in the current frame (e.g., position, size, appearance feature values), and outputs the target's track. Here, the track means time series data in which the feature values ​​of the target are arranged in a chronological order, and more specifically, refers to time series data in which the filtered feature values ​​after filtering, which will be described later, are arranged in a chronological order. In particular, when the target is a person, the observed value may be a bounding box in which the whole body of the person (first region) is the candidate region, or a bounding box in which the head of the person (second region) is the candidate region.

[0018] More specifically, as shown in FIG. 2, the tracker 12 includes a predictor 121, a correlator 122, and a filter 123.

[0019] (Prediction section) The prediction unit 121 predicts a feature amount at the current time (second time; current frame) from a feature amount at a past time (first time; previous frame) based on the feature amount output from the detection unit 11. The prediction unit 121 outputs the prediction result to the correlation unit 122 as a predicted feature amount.

[0020] Furthermore, the prediction unit 121 acquires an observed feature amount that is a feature amount at the current time detected and output by the detection unit 11 , and outputs the acquired observed feature amount to the correlation unit 122 .

[0021] (Correlation part) The correlation unit 122 compares the predicted features and observed features output from the prediction unit 121, determines a combination of the predicted features and observed features, and outputs the combined set of predicted features and observed features to the filter unit 123.

[0022] (Filter section) The filter unit 123 performs filtering using the pair of predicted feature amounts and observed feature amounts output from the correlation unit 122, and outputs the filtered feature amounts to the track classification unit 13 and the prediction unit 121. Here, the filtering may be a simple method such as an αβ filter, or a time series filter method based on statistical estimation such as a Kalman filter or a particle filter.

[0023] (Track Classification Department) The track classification unit 13 classifies tracks based on the track classification parameters and track information, which is time-series data of filtered features obtained from the tracking unit 12, counts the tracks classified for each attribute using the track classification parameters, and outputs the classified tracks and track information such as the number of counted tracks to the track processing unit 14 or the display unit 3. The track classification parameters are stored in the memory unit 2, and the track classification unit 13 acquires the track classification parameters from the memory unit 2.

[0024] (Wake processing department) The track processing unit 14 is a functional unit that processes the track to be displayed based on the track information output from the track classification unit 13. That is, when displaying individual track information (e.g., a person's head, etc.), the track processing unit 14 may process the track in consideration of privacy or to control information. For example, the track processing unit 14 may apply mosaic processing or blacking out to an area based on a track that has been tracked (e.g., an area where no detection result exists and is substituted by prediction). Alternatively, the track processing unit 14 may apply mosaic processing or blacking out to tracks of a specific attribute classified by the track classification unit 13.

[0025] (Display) The display unit 3 displays track statistical information, individual track information, or individual processed or unprocessed tracks based on the output from the track classifier 13 or the track processor 14.

[0026] <Operation> Next, the operation of the target tracking device 10 will be described with reference to Figures 3A and 3B. Figure 3A is a flowchart relating to the track classification learning process, and Figure 3B is a flowchart relating to the track classification inference process.

[0027] (Wake classification learning process) The wake classification learning process is a process for calculating wake classification parameters based on past data. To perform such a process, the wake classification learning process includes an object detection process (step ST11), an object tracking process (step ST12), an annotation process (step ST13), and a parameter estimation process (step ST14).

[0028] First, in step ST11, an object detection process is performed. More specifically, in step ST11, the detection unit 11 detects a specific part of an object from raw data such as an image for each frame obtained from the sensor observation unit 1, and calculates a feature amount of the detected part. In the case where the object is a person, the specific part of the object is the head or the whole body of the person.

[0029] Next, in step ST12 following step ST11, the tracking unit 12 performs object tracking processing. More specifically, it is as follows. The prediction unit 121 predicts a feature amount at the current time from a feature amount at a past time output from the detection unit 11. The correlation unit 122 compares the predicted feature amount output from the prediction unit 121 with a feature amount at the current time, determines a combination of the predicted feature amount and the observation feature amount at the current time, and outputs a feature amount to be filtered to the filter unit 123. The filter unit 123 filters using the predicted feature amount at the current time and the observation feature amount at the current time, and outputs the filtered feature amount to the track classification unit 13 and the prediction unit 121. If there is no observation feature amount to be correlated, the predicted feature amount is output to the track classification unit 13 as a filtered feature amount. The filtered feature amount includes feature amounts such as the position and size of the target. The tracker 12 outputs, in addition to the filtered feature quantity, tracking quality information such as error covariance, correlation count, nearby track information, and the presence or absence of a memory track indicating a track when there is no correlation, calculated in the tracker 12, to the track classification unit 13. The error covariance is calculated by the filter unit 123 using, for example, a Kalman filter. The correlation count is calculated as the number of times correlation is performed by the correlation unit 122. The nearby track information is information indicating tracks nearby to the tracking track. The nearby track information is obtained by the correlation unit 422 calculating the distance between tracks.

[0030] Next, in step ST13 following step ST12, annotation processing is performed. More specifically, in step ST11, the track classification unit 13 performs labeling on tracks having specific attributes, based on the position information of the track and data information of the track, such as video information or intensity information, for the track output from the tracking unit 12.

[0031] Classification by location information can be based on factors that are considered regarding the presence of targets in the sensing area, such as the appearance location, the destination (disappearance) location, the duration of stay, or the adjacency of the tracks. Track classification may be performed taking multiple factors into consideration. Tracks to be classified may be classified into any of the following multiple movement patterns: (Movement pattern 1) A wake that appeared from the right as seen from the sensor (Movement pattern 2) A wake that appeared from the left as seen from the sensor (Movement pattern 3) A wake that appeared from the back and disappeared in front as seen by the sensor (Movement pattern 4) A wake that appeared from the front and disappeared into the distance as seen by the sensor (Movement pattern 5) A hurried trail that disappears quickly despite the long distance between appearance and disappearance (Movement pattern 6) A wake that is interested in objects in the sensing area that stay in the sensing area for a long time (Movement pattern 7) A track belonging to multiple tracks of multiple targets that show similar changes (group track pattern). For example, a track that is close to other tracks that have a specific positional relationship such as translation and has a speed similar to that of the other tracks.

[0032] Regarding labeling of specific attributes, for example, when determining the attribute of the presence or absence of the action of gazing at an object of interest such as a notice or display, among the track data, a track in which the object of interest was gazed at is labeled as a gaze track. Alternatively, among the tracks, a track corresponding to the time period of gaze is extracted and labeled as a gaze track. Moreover, a track that does not correspond to a gaze track is labeled as a non-gaze track. Here, when labeling, low-quality tracks are excluded from classification based on tracking quality information, or classification is performed by attaching a label indicating low quality.

[0033] FIG. 7 shows the track when there are two targets, a sensor, and an object of interest such as a poster. A or P BThe lines indicated by are the position information of the wake, and the small square frames on the wake indicate the image data (head image) of the wake within the detection area obtained at that point. The part of the wake that is gazing at the object of interest is called the gaze wake. In Figure 7, the gaze wake is indicated by a large square frame. Also, a wake in which the filtered images or predicted positions of target A and target B are close to each other is called a close wake, and since the quality of the wake is low when the targets are close to each other, we make an effort not to use such close wakes for gaze judgment.

[0034] Next, in step ST14 following step ST13, a parameter estimation process is performed. In step ST14, the track classification unit 13 calculates parameters for track classification based on the tracks and the labeled label data for each movement pattern, and stores the calculated parameters in the storage unit 2 as track classification parameters.

[0035] For example, the track classification unit 13 uses vector X, which is an arrangement of feature values ​​obtained by calculating HOG (Histograms of Oriented Gradients), which is a gradient histogram, for head image data included in the track, as the feature value of the track, and estimates learning parameters using a learning method such as linear discriminant analysis using vector X and label data. In linear discriminant analysis, parameters such as a matrix W, which is an arrangement of eigenvectors projecting the original feature values, an average vector of the projected feature values ​​for each class, and a standard deviation are calculated as learning parameters. The track classification unit 13 saves the calculated parameters in the memory unit 2 as track classification parameters, and uses the saved track classification parameters in the track classification inference process. In this way, the track classification unit 13 learns track classification parameters for classifying multiple movement patterns based on the filtered image.

[0036] (Wake classification inference processing) The wake classification inference process is a process of classifying wakes using the wake classification parameters obtained by the wake classification learning process. To perform such a process, the wake classification inference process includes an object detection process (step ST21), an object tracking process (step ST22), and a wake classification process (step ST23). The object detection process (step ST21) and the object tracking process (step ST22) are similar to the object detection process (step ST11) and the object tracking process (step ST12) in the wake classification learning process, respectively, and therefore overlapping explanations will be omitted.

[0037] In step ST23, a track classification process is performed. The track classification process is performed by the track classification unit 13. First, the tracks obtained from the tracking unit 12 are classified based on the position information of the tracks. In the track classification process, when the tracking unit 12 continues to have no detection results that correlate with the predicted track and the track disappears, or when a track of a certain length is generated, the track output from the tracking unit 12 selects a movement pattern that is most similar to the track among the movement patterns classified in the track separation learning process. For example, the similarity of the tracks or the proximity of the start point or end point (vanishing point) of the track may be used as a criterion for the selection of the movement pattern. In addition, a determination is made as to whether the target track has performed a specific action (gazing) using the track classification parameters for each selected movement pattern. Specifically, first, a feature amount is calculated by arranging the feature amount calculated by HOG for the image data of the track in a chronological order. Next, the statistical distance is calculated from the projected feature obtained by projecting the calculated feature with a projection matrix in which eigenvectors, which are track classification parameters, and the average vector and standard deviation for each class, the class with the smallest statistical distance is selected, and the action represented by the selected class is output to the display unit 3. For example, when determining whether or not a person is gazing as a specific action, the tracks belonging to the gazing class may be counted and the number of counted tracks may be displayed as the number of people gazing on the display unit 3, or the video data may be displayed and the tracks determined to be gazing may be highlighted at the same time. The track classification process may be performed when a stable track disappears.

[0038] In a step (not shown) following step ST23, the track processor 14 may process the track.

[0039] Next, referring to Fig. 9 and Fig. 10, an example of the hardware configuration of the target tracking device 10 and the target tracking system including the target tracking device 10 will be described. The functions of the target tracking device 10 are realized by a processing circuitry. The processing circuitry may be a dedicated processing circuit 102a as shown in Fig. 9, or a processor 102b that executes a program stored in a memory 102c as shown in Fig. 10. In addition, the sensor observation unit 1 and the display unit 3 included in the target tracking system are realized by, for example, a camera 101 and a display 104, respectively.

[0040] When the processing circuitry is a dedicated processing circuit 102a, the dedicated processing circuit 102a may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of these. The functions of the target tracking device 10 may be realized by a plurality of separate processing circuits, or the functions may be collectively realized by a single processing circuit. In addition, a memory (not shown) is connected to the processing circuit 102a to realize the storage unit 2.

[0041] When the processing circuitry is the processor 102b, the functions of the target tracking device 10 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 102c. The processor 102b realizes the functions of each functional unit of the target tracking device 10 by reading and executing the programs stored in the memory 102c. Here, examples of the memory 102c include non-volatile or volatile semiconductor memories such as random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, mini disks, and DVDs.

[0042] It should be noted that some of the functions of the target tracking device 10 may be realized by dedicated hardware, and some by software or firmware. In this way, the processing circuit can realize each of the above-mentioned functions by hardware, software, firmware, or a combination of these.

[0043] According to the target tracking device 10, by analyzing the video image information for each movement pattern of the wake, it is possible to mitigate the influence of different gaze purposes or gaze methods depending on the movement pattern, and to grasp the detailed movement of the wake with high accuracy. In addition, by using the image information based on the filtered feature values ​​output from the tracking unit 12, and by performing labeling based on the tracking quality, it is possible to classify the data of the scene where it is difficult to distinguish the target separately, so that it is expected to improve the accuracy of the movement estimation.

[0044] Embodiment 2 A target tracking device according to the second embodiment of the present disclosure will be described with reference to FIG. 4. FIG. 4 is a diagram showing a configuration example of a target tracking device 20 according to the second embodiment. As shown in FIG. 4, the target tracking device 20 includes a detection unit 21 and a tracking unit 12, similar to the target tracking device 10 according to the first embodiment. The target tracking device 20 includes a direction estimation unit 24 as an additional functional unit, and a track classification unit 23 in which the function of the track classification unit 13 is changed in response to the addition of the direction estimation unit 24. Also, similar to the first embodiment, the target tracking device 20 may include a track processing unit (not shown). Also, as shown in FIG. 4, a target tracking system using the target tracking device 20 includes a sensor observation unit 1, the target tracking device 20 connected to the sensor observation unit 1, a storage unit 2 connected to the target tracking device 20, and a display unit 3 connected to the target tracking device 20.

[0045] (Orientation Estimation Unit) The orientation estimation unit 24 estimates orientation estimation parameters, which are parameters for estimating orientation such as azimuth or elevation, from past tracks stored in the memory unit 2, i.e., time-series data of filtered images, positions, and track quality. The orientation estimation unit 24 infers the orientation of an object using the estimated orientation estimation parameters for the currently obtained track.

[0046] The orientation estimation unit 24 subdivides the orientation of the object and annotates the corrected image output from the tracking unit 12 for each orientation of the object, i.e., the image based on the filtered feature amount. For example, the azimuth and elevation of the object are subdivided, and the corrected image for each specific value of the azimuth and elevation is annotated. The orientation estimation unit 24 also learns orientation estimation parameters, and outputs the result of estimating the orientation of the currently obtained track to the track classification unit 23.

[0047] The track classification unit 23 judges whether the target has gazed at the object of interest based on the orientation information and the position of the track inferred by the orientation estimation unit 24, i.e., based on the angle between the relative position of the object of interest and the track and the orientation estimated by the orientation estimation. The gaze judgment is performed, for example, by using an N-out-of-M judgment method to judge whether the angle is the angle at which the object of interest was viewed N times out of the past M times. When making this gaze judgment, images with degraded quality, for example images with a high possibility of degrading the detection result such as the presence of a nearby track, may not be used as targets for the N-out-of-M judgment based on the track quality.

[0048] The classification of the tracks may be performed using a likelihood ratio L according to the following formula (1). That is, the track classification unit 23 may calculate a likelihood ratio, which is a ratio between the probability that the tracking track is a gaze track and the probability that the tracking track is a normal track that is not a gaze track, from the position of the tracking track and the direction of the target, and infer the probability that the target gazed at the object of interest from the magnitude of the calculated likelihood ratio. In formula (1), H1 indicates a specific action hypothesis, for example, gaze or suspicious / abnormal behavior, and H0 indicates a normal track. p(HP,P,RP|H1) indicates the probability that the target is a gaze track that performed a specific action such as gaze, and p(HP,P|H0) indicates the probability that it is a normal track. HP is a vector in which the direction (azimuth) estimated from an image of one snapshot is arranged in time series, P is a vector in which the center or foot position of the target is arranged in time series, and RP is the position vector of the object of interest such as a notice. L=p(HP,P,RP|H1) / p(HP,P|H0) (1)

[0049] p(HP,P,RP|H1) is a probability distribution of an exponential distribution family such as a normal distribution that has a high probability when there is a high degree of match between the direction HP estimated from the image and the direction of the position vector based on the object of interest calculated from the difference between P and RP, and when the difference between P and RP is within a certain distance within the field of view, and p(HP,P|H0) is a probability distribution of an exponential distribution family such as a normal distribution that has a high probability when the direction of the velocity vector, which is the time difference of P, matches the direction HP calculated from the image, assuming that this is a normal action in which the movement and direction match.

[0050] By outputting the trails that exceed the threshold Th for the likelihood ratio (L>Th) as specific trails, it is possible to extract trails that have a high probability of performing a specific action and a low probability of performing a normal action. Here, Th may be determined using a likelihood ratio test or the like, or the threshold Th may be adjusted using existing correct answer data so that the probability of misrecognition is constant.

[0051] If there is no particular poster or other object of interest and it is desired to determine whether the wake is abnormal, then instead of using the likelihood ratio, the inverse L2 of the probability that the wake is normal p(HP,P|H0) can be calculated as shown in the following equation (2), and the abnormal wake can be classified using a threshold judgment process (L2>Th2). L2=1 / p(HP,P|H0) (2)

[0052] By using L2 to perform threshold judgment, it is possible to extract people whose facial direction in the wake does not match the direction of the wake's position vector.

[0053] As is clear from the above, according to this second embodiment, it is possible to estimate detailed movement of a wake by classifying the wake from orientation information obtained through image processing, position information of the wake, and the position vector of the object of interest.

[0054] Embodiment 3 A target tracking device according to the third embodiment of the present disclosure will be described with reference to FIG. 5 and FIG. 8. FIG. 5 is a diagram showing a configuration example of the target tracking device 30 according to the third embodiment, and FIG. 8 is a diagram for explaining the operation of the target tracking device 30. As shown in FIG. 5, the target tracking device 30 includes a detection unit 31, a tracking unit 32, and a track classification unit 13, similar to the target tracking device 10 according to the first embodiment. In addition, the target tracking device 30 includes a position estimation unit 34 as a new functional unit. With the new provision of the position estimation unit 34, the detection unit 31, the tracking unit 32, and the track classification unit 13 of the target tracking device 30 are improved from the detection unit 11, the tracking unit 12, and the track classification unit 13 of the target tracking device 10, respectively. In addition, similar to the first embodiment, the target tracking device 30 may include a track processing unit (not shown). The improvements will be described below.

[0055] In the first embodiment, the detection unit 11 detects the head of a person. In the third embodiment, the detection unit 31 detects not only the head of a person but also the whole body of the person. More generally, the detection unit 31 detects a first region (whole body) of a target and a second region (head) narrower than the first region. The detection unit 31 outputs the feature amount of the whole body of the person to the tracking unit 32 in addition to the feature amount of the head of the person.

[0056] The tracking unit 32 outputs the track of the first region and the track of the second region as the tracking track. That is, the tracking unit 32 tracks the head and the whole body, and outputs the tracking track to the position estimation unit .

[0057] The position estimation unit 34 estimates the position of the wake in the first area and the position of the wake in the second area, determines the correspondence between the wake in the first area and the wake in the second area, and calculates the second position of the target. This will be specifically described below.

[0058] The position estimation unit 34 correlates the head trail and the whole body trail based on the relationship between their relative positions and sizes, and calculates the distance to the head from the foot position of the whole body trail correlated with the internal and external parameters of the camera. This allows the head distance to be accurately calculated based on the foot position. As an example of the correlation between the head trail and the whole body trail, the correspondence (correlation) between the trails of head detection and whole body detection is determined based on the relative positions of the head detection and the whole body detection as shown in FIG. 8. Here, taking into consideration detection failure areas such as in crowded environments where only head detection is effective, the size of the target's whole body may be estimated based on trail information at the time when head detection and whole body detection appear simultaneously, and used for distance calculation. The position estimation unit 34 sends the calculated position of the head trail to the trail classification unit 33.

[0059] The track classification unit 13 classifies the tracks based on the accurate head position information calculated above.

[0060] As is clear from the above, according to this embodiment 3, it is possible to obtain accurate position information of the head wake, and therefore it is possible to accurately classify the wake based on the accurate position information.

[0061] Embodiment 4 A target tracking device according to a fourth embodiment of the present disclosure will be described with reference to Fig. 6. Fig. 6 is a diagram showing a configuration example of a target tracking device 40 according to the fourth embodiment. As shown in Fig. 6, the target tracking device 40 includes a detection unit 41, a tracking unit 42, and a track classification unit 43, similar to the target tracking device 10 according to the first embodiment. In addition, the target tracking device 40 includes a motion correction unit 44 as a new functional unit. In addition, similar to the first embodiment, the target tracking device 40 may include a track processing unit (not shown).

[0062] The motion correction unit 44 is a functional unit that outputs motion information for correcting a misalignment between a first image at a first time and a second image at a second time immediately after the first time due to a difference between a first position where the first image was acquired and a second position where the second image was acquired. That is, the motion correction unit 44 corrects a misalignment between the first image and the second image when the sensor observation unit 1 acquires the first image and the second image at different positions.

[0063] In order to estimate the motion of the entire image, the motion compensation unit 44 applies a feature point extraction method such as ORB (Oriented FAST and Rotated BRIEF) or AKAZE (Accelerated KAZE) and a feature point matching method to two images from consecutive frames, and estimates an average vector of the movement of the entire image between frames, or an affine transformation matrix or a homography transformation matrix of the image from the obtained feature point matching result. The motion compensation unit 44 outputs the estimation result to the tracking unit 42 and the wake classification unit 43 as image motion information. Here, when correcting the motion of the entire image, the feature point extraction method may be applied to outside the detection candidate area by the detection unit 41 so as not to include speed components due to individual moving objects. In addition, the amount of movement on the two-dimensional bird's-eye coordinate system may be calculated from the image motion information using the internal parameters and external parameters of the camera.

[0064] The tracking unit 42 converts the predicted motion vector of the individual target using the motion information of the image input from the motion correction unit 44, correcting the predicted vector to track the target. For example, the average vector of the overall motion of the image is subtracted from the predicted motion vector of the individual target. By performing the correction in this manner, it becomes possible to accurately predict the position of each individual target.

[0065] The track classification unit 43 uses the image motion information input from the motion correction unit 44 to correct the track position information and make a judgment.

[0066] Here, the motion correction unit 44 may estimate image motion information or the amount of movement of the sensor using self-position estimation information from an internal sensor such as an INS (Inertial Navigation System) sensor.

[0067] As is clear from the above, according to this embodiment 4, by assuming that the platform (not shown) on which the sensor observation unit 1 is mounted moves, and correcting the sensing data taking into account the movement of the sensor observation unit 1, it is possible to accurately track targets and classify their tracks.

[0068] It is possible to combine the embodiments, and to modify or omit each embodiment as appropriate. [Industrial Applicability]

[0069] The target tracking device of the present disclosure can be used as a device for tracking individual targets for various purposes, such as preventive maintenance or guiding advertising, in an environment where multiple targets are present. [Explanation of symbols]

[0070] 1 sensor observation unit, 2 memory unit, 3 display unit, 10 target tracking device, 11 detection unit, 12 tracking unit, 13 track classification unit, 14 track processing unit, 20 target tracking device, 21 detection unit, 23 track classification unit, 24 orientation estimation unit, 30 target tracking device, 31 detection unit, 32 tracking unit, 33 track classification unit, 34 position estimation unit, 40 target tracking device, 41 detection unit, 42 tracking unit, 43 track classification unit, 44 motion compensation unit, 101 camera, 102a processing circuit, 102b processor, 102c memory, 104 display, 121 prediction unit, 122 correlation unit, 123 filter unit.

Claims

1. A detection unit that detects a feature amount including a position of at least one target from the sensor data; a tracking unit that tracks the target based on the detected feature amount and outputs a tracking trail of the target to be tracked, the tracking trail including position information of the tracking trail and data information of the tracking trail; a track classification unit that classifies the output tracking track into one of a plurality of predetermined movement patterns based on a plurality of consideration factors related to the manner in which the target exists in a sensing area, which is indicated by position information and data information of the tracking track, the plurality of consideration factors including at least one of a position, a speed, a relationship with other tracks, and an orientation of the target, and outputs track information of the classified tracking track; A target tracking device comprising:

2. A target tracking device as described in claim 1, wherein the data information includes at least moving image information or intensity information.

3. A target tracking device as described in claim 1, wherein the positions in the multiple consideration factors include an appearance position and a disappearance position within the sensing area.

4. The tracking unit is a prediction unit that predicts a predicted feature amount at a second time later than a first time from a first observed feature amount detected at the first time; a correlation unit that determines a correlation between the predicted feature and a second observed feature detected at the second time; a filter unit that performs filtering using the correlated second observed feature amount and the predicted feature amount, and outputs time series data of the filtered feature amount as the tracking wake; Equipped with 2. A target tracking device according to claim 1.

5. the sensor data is an image; The track classification unit learns track classification parameters for classifying the plurality of predetermined movement patterns based on the filtered image.

5. A target tracking device according to claim 4.

6. the track classification unit uses tracking quality information indicating quality of the track to determine which of a plurality of predetermined movement patterns the output tracking track corresponds to; 2. A target tracking device according to claim 1.

7. Further comprising a track processing unit that processes the content to be displayed based on the output track information.

2. A target tracking device according to claim 1.

8. a direction estimation unit that estimates a direction of the target estimated from the image; 6. The target tracker of claim 5, further comprising:

9. The track classifier infers whether the target is gazing at the object of interest based on a degree of agreement between a position of the target relative to a position of an object of interest and the estimated orientation of the target.

9. A target tracking device according to claim 8.

10. the track classification unit calculates a likelihood ratio, which is a ratio between a probability that the tracking track is a gaze track and a probability that the tracking track is a normal track other than the gaze track, from the position of the tracking track and the estimated direction of the target, and infers a probability that the target has gazed at the object of interest from the magnitude of the calculated likelihood ratio.

9. A target tracking device according to claim 8.

11. the track classification unit infers whether the target is behaving suspiciously or abnormally based on a degree of agreement between a direction of a velocity vector, which is a time change in the position of the target, and the estimated direction of the target; 9. A target tracking device according to claim 8.

12. The track classification unit performs the determination when the tracking track disappears.

2. A target tracking device according to claim 1.

13. The detection unit detects a first area of ​​the target and a second area narrower than the first area, the tracking unit outputs a track of the first area and a track of the second area as the tracking track, a position estimating unit that estimates a position of a track in the first area and a position of a track in the second area, determines a correspondence relationship between the track in the first area and the track in the second area, and calculates the second position of the target. A target tracking device according to any one of claims 1 to 12.

14. the at least one target is a plurality of targets; The predetermined plurality of movement patterns includes a group track pattern in which a plurality of tracks show similar changes, the track classification unit determines that a tracking track of any one of the plurality of targets corresponds to the group track pattern; 2. A target tracking device according to claim 1.

15. The sensor data is a plurality of images successive in time, a motion correction unit that outputs motion information for correcting a deviation between a first image at a first time and a second image at a second time immediately after the first time due to a difference between a first position at which the first image is acquired and a second position at which the second image is acquired, The tracking unit uses the output motion information to predict the position of each target excluding the movement of the sensor or the overall movement of the image, and tracks the target.

2. A target tracking device according to claim 1.

16. The sensor data is a plurality of images successive in time, a motion correction unit that outputs motion information for correcting a deviation between a first image at a first time and a second image at a second time immediately after the first time due to a difference between a first position at which the first image is acquired and a second position at which the second image is acquired, The track classification unit uses the output motion information to correct the position information included in the output tracking track to position information of individual tracks excluding the movement of the sensor or the overall movement of the image, and performs the determination.

2. A target tracking device according to claim 1.

17. A detection unit that detects a feature amount including a position of at least one target from sensor data; a tracking unit that tracks the target based on the detected feature amount and outputs a tracking trail of the tracked target; a track classification unit that classifies the output tracking track into one of a plurality of predetermined movement patterns and outputs track information of the classified tracking track; Equipped with the sensor data is an image; The track classification unit learns track classification parameters for classifying the plurality of predetermined movement patterns based on the filtered image. Target tracking device.

18. A detection unit that detects a feature amount including a position of at least one target from sensor data; a tracking unit that tracks the target based on the detected feature amount and outputs a tracking trail of the tracked target; a track classification unit that classifies the output tracking track into one of a plurality of predetermined movement patterns and outputs track information of the classified tracking track; Equipped with The detection unit detects a first area of ​​the target and a second area narrower than the first area, the tracking unit outputs a track of the first area and a track of the second area as the tracking track, a position estimating unit that estimates a position of a track in the first area and a position of a track in the second area, determines a correspondence relationship between the track in the first area and the track in the second area, and calculates the second position of the target. Target tracking device.

19. A target tracking method performed by a target tracking device having a detection unit, a tracking unit, and a track classification unit, comprising: A step of detecting a feature amount including a position of at least one target from the sensor data by the detection unit; a step of tracking the target based on the detected feature amount by the tracking unit, and outputting a tracking trail of the target being tracked, the tracking trail including position information of the tracking trail and data information of the tracking trail; a step of the track classification unit determining which of a plurality of predetermined movement patterns the output tracking track corresponds to based on a plurality of consideration factors regarding the manner in which the target exists in a sensing area indicated by position information and data information of the tracking track, the plurality of consideration factors including at least one of a position, a speed, a relationship with other tracks, and an orientation of the target; A target tracking method comprising:

Images