Object tracking device and program
By focusing on stable feature points and intermittent object detection, the system addresses tracking accuracy and computational issues, achieving efficient and accurate object tracking with reduced occlusion.
Patent Information
- Application Number
- JP2021096895
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-06-09
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-06-09
AI Technical Summary
Conventional object tracking technologies face issues with tracking accuracy due to feature point transfer to other objects during occlusion and high computational requirements, especially when using motion vectors and lacking statistical machine learning elements for object detection.
The system tracks object areas using feature points like joint points above the chest, such as shoulders and head, and intermittently applies object detection to correct tracking results, utilizing a combination of tracking units and correction sections to stabilize tracking by minimizing occlusion and maintaining efficient calculation.
This approach enhances tracking accuracy and efficiency by stabilizing tracking with minimal occlusion and reducing computational load, allowing for precise and fast object tracking in various scenarios.
Smart Images

Figure 0007584359000001 
Figure 0007584359000002 
Figure 0007584359000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an object tracking device and a program for tracking an object from a video. [Background technology]
[0002] Patent Documents 1 and 2 are examples of conventional techniques for tracking objects from video. In Patent Document 1, in order to achieve robust tracking, tracking is performed by combining a motion vector (optical flow) that can capture the detailed movement of an object with object detection that can estimate the exact position of the object. Since object detection takes time, it is performed only when the reliability of tracking has decreased, thereby achieving high speed. In Patent Document 2, in order to deal with objects passing each other or sudden changes in movement, tracking is performed by combining candidate positions estimated at long intervals by a particle filter with candidate positions estimated at short intervals by a motion vector (optical flow). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Special Publication No. 2018-523877 [Patent Document 2] JP 2017-076181 A Summary of the Invention [Problem to be solved by the invention]
[0004] The conventional techniques have the following problems. In Patent Document 1, feature points that are the source of a motion vector are acquired at uniform intervals throughout the entire tracking rectangle. Therefore, when occlusion occurs, feature points are likely to move to other objects, and there are problems with tracking accuracy. In addition, a large number of feature points are required, and there are also problems with calculation time. In Patent Document 2, since there is no element of statistical machine learning such as a trained object detector, there are problems with object detection accuracy. Therefore, tracking based on this detection also has problems with tracking accuracy.
[0005] In view of the above problems in the conventional technology, an object of the present invention is to provide an object tracking device and program capable of tracking an object more efficiently than with video. [Means for solving the problem]
[0006] In order to achieve the above-mentioned object, the present invention provides an object tracking device comprising: a tracking unit that continuously tracks an object region, which is a region of an object imaged in a frame, across frames of a video at different times; and a correction unit that corrects the object region tracked by the tracking unit for a portion of frames in the video and causes the tracking unit to continue tracking the corrected object region, wherein a first feature of the correction unit is that it detects objects from the portion of frames and performs the correction by associating each of the detected object regions with each of the object regions tracked by the tracking unit.
[0007] The second feature is that the tracking unit tracks the object region by tracking feature points of the object, each of the objects is an individual person, the feature points of the object tracked by the tracking unit include joint points that are higher than the chest of the person, such as joint points of the head and shoulders, and do not include points located below the chest, and the object region tracked by the tracking unit is a region over the entire body of the person. The third feature is that the tracking unit tracks the object region by tracking feature points of the object, the feature points of the object tracked by the tracking unit include only predetermined types of joint points that are expected to have little change in the positional relationship of the joint points of the person, such as the head, shoulders, torso, or buttocks, and the object region tracked by the tracking unit is a region over the entire body of the person. The third feature is that the object tracking device is a program that causes a computer to function as an object tracking device according to the first to third features. Effect of the Invention
[0008] According to the first feature, the tracking unit continues object tracking while the correction unit intermittently applies object detection to some frames in the video to correct the object tracking results, thereby enabling efficient object tracking from the video. According to the second or third feature, when tracking the entire body of a person, the tracking is stabilized by tracking only points that are expected to be less occluded and have stable movement depending on the situation, thereby enabling efficient object tracking from the video. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a functional block diagram of an object tracking device according to an embodiment. [Diagram 2] 1 is a flowchart of an operation of an object tracking device according to an embodiment. [Diagram 3] FIG. 11 is a diagram showing a schematic example of processing of an object tracking device. [Figure 4] FIG. 11 is a diagram showing a schematic example of processing of an object tracking device. [Diagram 5] FIG. 11 is a diagram showing a schematic example of processing of an object tracking device. [Figure 6] FIG. 11 is a diagram showing a schematic example of processing of an object tracking device. [Figure 7] FIG. 1 is a diagram illustrating a hardware configuration of a typical computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] 1 is a functional block diagram of an object tracking device 10 according to an embodiment. As shown in the figure, the object tracking device 10 includes an imaging unit 1, an initialization unit 2, a tracking unit 3, an object detection necessity determination unit 4, a correction unit 5, and an output unit 6. Furthermore, the initialization unit 2 includes a first detection / extraction unit 21, a tracking start determination unit 23, and a first feature point detection unit 24, the tracking unit 3 includes a motion vector estimation unit 31, a range estimation unit 32, and a tracking end determination unit 33, and the correction unit 5 includes a second detection / extraction unit 51, a range correction unit 52, a tracking start / end determination unit 53, and a second feature point detection unit 54.
[0011] Fig. 2 is a flowchart of the operation of the object tracking device 10 according to an embodiment. The overall outline of this flow is that steps S1 to S6 are repeatedly executed at each time as a process of detecting an object from a frame of a video at each time and then tracking the object. Below, the details of the process of each functional block in Fig. 1 will be explained while explaining each step in Fig. 2. Figs. 3 to 6 are diagrams respectively showing schematic examples of the process of the object tracking device, divided into examples EX1 to EX10, and will be referred to as appropriate in the following explanation.
[0012] First, a problem setting for object tracking executed by the object tracking device 10 will be described, along with an explanation of variable notations used below. Example EX1 in Fig. 3 shows how object tracking works.
[0013] In the example EX1, two objects (e.g., people) are present in two adjacent frames at time t and t+1. t =(b t 1 , b t 2 , …) for frame o at time t (t=1,2,…). t (The frame time t is a positive integer and is expressed as a frame number.) Here, b t i Frame O t This indicates the i-th (i=1, 2, ...) object range in the frame. The object range can be expressed as a rectangle (enclosing frame, bounding box) such as b=(x, y, w, h) in the image coordinate system of the frame, where x and y are the x and y coordinates of the upper left corner of the rectangle and indicate the position of the range (the position of a representative point such as the upper left), and w and h are the width and height of the rectangle and indicate the size of the range. t The i-th object range in b t i For object ID (identifier), z t i Add a to a t i =(b t i ,z ti ) and frame them as t The items you collect are A t =(a t 1 ,a t 2 The object ID is found by matching objects between frames at different times in the video. Object matching is done using features such as the appearance and position of the object. In other words, object tracking is a process of tracking a time series of images (frames) O={o t Given |t ≥ 1}, Ω = {A t |t ≧ 1} (the problem of tracking the object range while identifying the ID of each object from the frame image). Therefore, in order to achieve highly accurate object tracking, it is necessary to accurately determine both the object range and the object ID.
[0014] <Step S1...imaging unit 1> 2 starts, the process proceeds to step S1. Alternatively, as shown in FIG. 2, when steps S6 and S61 are executed, the process returns to step S1. In step S1, the image capturing unit 1 captures an object and outputs a frame (image) o at the current time t. t After obtaining the above, the process proceeds to step S10. The imaging unit 1 is constituted by a camera as hardware, and can obtain frames by capturing images.
[0015] Frame o captured by the imaging unit 1 tis output to the first detection / extraction unit 21 of the initialization unit 2 when it is subject to the initialization unit 2 (when steps S21, S23, and S24 are followed), is output to the motion vector estimation unit 31 of the tracking unit 3 when it is subject to the tracking unit 3 (when steps S31, S32, and S33 are followed), and is also output to the second detection / extraction unit 51 of the correction unit 5 as shown by line L51 in FIG. 1 when it is subject to the correction unit 5 in addition to the tracking unit 3 (when steps S31, S32, and S33 are followed and then steps S51, S52, S53, and S54 are followed from step S4). (Note that when it is subject to the correction unit 5, in addition to the output shown by line L51 in FIG. 1, the output shown by lines L52 and L5 is performed.) Also, although the flow of output is omitted in FIG. 1, the frame o of the current time t obtained by the imaging unit 1 is used by the output unit 6 to output the result for the current time t in step S6 described later. t may further be output to the output unit 6.
[0016] <Step S10...Branch depending on whether it is the initial time or not> In step S10, the object tracking device 10 determines whether the current time t is the initial time t=1 or not. If the determination is affirmative (if the initial time t=1), the process proceeds to step S21. t In steps S21, S23, and S24, the initialization unit 2 initializes frame o = o1. If the initialization is negative (if t ≥ 2 and it is not the initial time), the process proceeds to step S31. t are subjected to the tracing process by the tracing unit 3 in steps S31, S32, and S33.
[0017] The initial time t means a time when there is no frame image captured by the imaging unit 1 at past times t-1, t-2, ..., or when there is a frame image from the past time but object tracking has not been performed by the object tracking device 10 and past object tracking results cannot be referenced. (That is, the time when the object tracking device 10 starts capturing frame images and tracking objects from each frame is set as the initial time.) For the sake of explanation, this initial time is set as t=1.
[0018] <Step S21...First detection and extraction unit 21> In step S21, the first detection and extraction unit 21 detects frame o at the initial time t=1. t = o1 image, each object range D t =(d t 1 ,d t 2 ,…)(t=1). (Note that in this range D t is the range B shown in the example EX1 in Figure 3. t The position and size of the rectangle defined in the same way as above can be detected, but the final tracking target range B t Since is determined through the processing of the tracking start determination unit 23 and the first feature point detection unit 24 in the subsequent stage, in order to clearly indicate that it is a range as an intermediate result, the variable notation is differentiated and the range D t ) For each of these objects, the tracking features are extracted as targets that can be used for tracking at a later time t≧2. The object position detection and feature extraction can be performed using existing methods, and can be performed simultaneously using a multitask deep neural network disclosed in the following Non-Patent Document 1. [Non-patent document 1] Y. Zhang, C. Wang, X. Wang, W. Zeng, and W. Liu, "FairMOT: On the fairness of detection and re-identification in multiple object tracking," Computing Research Repository arXiv Preprint arXiv:2004.01888, 2020.
[0019] In one embodiment, the features extracted by the first detection / extraction unit 21 can be used when performing correspondence based on features in a range correction unit 52 of the correction unit 5, which will be described later. In Fig. 1, the flow of output of features from the first detection / extraction unit 21 to the range correction unit 52 for this purpose is not depicted.
[0020] 3 shows a schematic example in which the first detection / extraction unit 21 detects an object range as a rectangular range within a frame and extracts its features (extracts features as vector quantities or tensor quantities) using a deep neural network. In step S21, the object range (and features) of each object in the frame at the initial time obtained by the first detection / extraction unit 21 is output to the tracking start determination unit 23 (and range correction unit 52), and then the process proceeds to step S23.
[0021] In addition, a multitask deep neural network can detect a single type or multiple types of objects from an image frame, and the type of object to be detected can be set in advance. Any type of object can be set as the detection target depending on the purpose of object tracking, and may be a living object such as a person or an animal, or a non-living object such as a robot or a car, but it is sufficient to set a moving object as the target of object tracking.
[0022] <Step S23...Tracking start determination unit 23> In step S23, the tracking start determination unit 23 detects the object range D t =(d t 1 ,d t 2 , …) (t=1), we determine whether to start tracking, and select only those objects that are determined to be in the starting position. t =(b t 1 , b t 2 , ...) (t=1). This determination of whether to start tracking may be made only for objects whose detection reliability when detected by the first detection and extraction unit 21 is equal to or higher than a predetermined threshold value. t i For b t i A t i =d t i Then, the object ID z is set to be unique. t iIn step S23, the object range B t =(b t 1 , b t 2 , …) (t=1) and their respective IDs z t i to the first feature point extraction unit 24, and then the process proceeds to step S24.
[0023] <Step S24...First feature point extraction unit 24> In step S24, the first feature point extraction unit 24 extracts the object range B t =(b t 1 , b t 2 , …)(t=1) for each b t i (i=1,2,…) to the characteristic point P t i =(p t i1 , p t i2 , …)(t=1) and detect this feature point P t i is output to the motion vector estimation unit 31 as a tracking target at the next time t=2, and the object range B t =(b t 1 , b t 2 , ...) (t=1) are output to the range estimation unit 32 as targets to be estimated at the next time t=2, and these feature points P t i and range B t (t=1) is output to the output unit 6 as the processing result of the initialization unit 2, and then the process proceeds to step S6. t ij HA t i It indicates the position of the jth (j=1,2,...) feature point in the image, and is expressed as a two-dimensional vector in the image coordinate system. t ij is the object ID z ti and ID is z t i The feature point set detected from an object is P t i ) To detect feature points, a detector that has been trained in advance can be used.
[0024] For example, if the tracking target is a person, the joint points estimated by the joint point detector can be used as the feature points. (Note that these joint points are skeletal joint points defined in the trained joint point detector, and do not necessarily have to match medical joints.) In this case, it is desirable to use, as feature points, not all of the predetermined types of joints detectable by the trained joint point detector, but a portion of the joints that are expected to have little change in position, such as points located above the chest, such as the head and shoulders (points whose altitude is equal to or higher than the chest when the person is standing). (That is, when considering a case where a person moves by walking or running, it is expected that the joints at the tips of the hands and feet will change their positions vigorously while vibrating in conjunction with the swinging of the hands and feet, and that they will be frequently obstructed by other objects because their altitude from the ground is relatively low, whereas the head and shoulders will change their positions relatively stably in conjunction with the center of gravity of the person, and will be less obstructed by other objects because their altitude from the ground is relatively high.)
[0025] The first feature point extraction unit 24 further detects a joint point as a feature point using a joint point detector, and then samples and uses not only one joint point but also multiple surrounding points as additional feature points, thereby improving robustness. (The sampled and added points are ultimately points that are not necessarily joints (points that are generally not joints, except when they coincide with some joint point by chance).)
[0026] The processing of the first feature point extraction unit 24 will be described by taking as an example a case where the head is used as a joint point that is expected to have little positional change. For example, M×K points (points defined radially distributed around the head joint) defined at positions on a circumference of a circle with a radius of m×r (m=1, 2, ..., M) and making an angle of 2π×(n / K) (n=0, 1, 2, ..., K-1) in radians are extracted from the frame o of the initial time t=1. t Multiple samples may be taken from the image of o1, and one joint point of the head may be added to the surrounding M × K points as feature points. Here, M and K are predetermined natural numbers. Also, r is the range b of the detected object as shown in the following formula (1). t i It is desirable to change the size of the filter based on the size (w, h). r=min(w,h) / L…Equation (1)
[0027] Here, L is a predetermined natural number. Furthermore, min(w,h) is a function that returns the shorter length of w and h, and for an object that is a vertically long rectangular range such as a person standing, min(w,h)=w. When additionally sampling points around the head joint, etc., these parameters M, K, and L can be set in advance so that points outside the range of the person's head, for example, background parts that are not people or points other than the person's head, are not sampled. Similarly, as surrounding points, it is also possible to additionally sample grid points in a checkerboard pattern within a certain range around the joint point, rather than radially around the joint point.
[0028] In example EX3 of Fig. 3, a schematic example is shown in which 2 x 8 = 16 surrounding points NB are added to one joint HD of the head, with M = 2 and K = 8 as above, to set them as feature points, and each feature point is depicted as a black circle (●). (Feature points will be similarly depicted as black circles in examples EX4 and onwards of Fig. 4 described later.) Although not shown in this example, if the joint points of the left and right shoulders are also detected in addition to the head, the surrounding points may be sampled for each of these shoulder joint points and added as feature points. In this case, different predetermined values may be used for parameters such as M, K, and L for the head and for the shoulders.
[0029] As described above, in one embodiment, the object to be tracked is a person, and the object range to be tracked is the range B that covers the whole body of the person. t While setting the above, the feature points that are actually to be tracked (the ones to be tracked by the motion vector estimation unit 31 described later) are limited to only the additional sampled points (the additional points also have an altitude of the person's chest or higher) of the head and shoulder joints and their surroundings, which are expected to have little occlusion and little change in positional relationship, and exclude points on the person whose altitude is lower than the person's chest (such as points on the person's legs) from the tracking targets, thereby making it possible to stabilize tracking by the motion vector estimation unit 31.
[0030] As a modified example of the above, the restriction that the altitude must be above the chest of a person may be relaxed to the vicinity of the torso or above (excluding arms and hands which move a lot). In other words, the object to be tracked is a person, and the object range to be tracked is a range B that covers the entire body of the person. tWhile setting the above, for the feature points that are actually tracked (targets tracked by the motion vector estimation unit 31 described later), only points sampled from a predetermined type of joint point (such as the head, shoulders, torso, or buttocks that are at an altitude higher than the torso) or the periphery of the joint point that are expected to have little change in positional relationship may be used. Since these points also have an altitude higher than the torso, it is expected that there will be little occlusion depending on the shooting conditions, and that there will be little change in positional relationship by moving in conjunction with the center of gravity of the person (when the person is walking normally and not performing extreme movements such as acrobatic movements), so tracking by the motion vector estimation unit 31 can be stabilized.
[0031] <Step S31...Motion Vector Estimation Unit 31> In step S31, the motion vector estimation unit 31 estimates the frame o at the previous time t−1. t-1 Feature point P in t-1 i =(p t-1 i1 , p t-1 i2 , …) is the frame o at the current time t. t A motion vector (optical flow) Δt indicating where the object moved i =(Δ t i1 ,Δ t i2 ,…), where Δ t ij Each optical flow is a two-dimensional vector that indicates the amount of change on the x-axis and y-axis. The motion vector can be estimated by an existing method such as the Lucas-Kanade (LK) method disclosed in the following Non-Patent Document 2. [Non-Patent Document 2] BD Lucas and T. Kanade, "An iterative image registration technique with an application to stereo vision," Proceedings of the 7th International Joint Conference on Articial Intelligence (IJCAI), pp.121-130, 1981.
[0032] When estimating a motion vector, the number of iterations of the algorithm and the convergence conditions are determined, and if a motion vector cannot be calculated even after calculation until the convergence is reached, the feature point is deleted as it is determined that the corresponding motion vector does not exist. For feature points that are not deleted because it is determined that a corresponding motion vector exists, the motion vector estimation unit 31 calculates the feature point P t-1 i The amount of movement is expressed as a motion vector Δt i Add the current feature point P t i By obtaining and storing the above-mentioned value in itself, the motion vector estimation unit 31 will again use the obtained value as the subject of motion vector estimation at the next time t+1 (if a negative judgment is made in step S4 described below and the correction unit 5 is not applied at the current time t, the motion vector estimation unit 31 will again use the obtained value as the subject of motion vector estimation at the next time t+1). P t i =P t-1 i +Δt i =(p t-1 i1 +Δ t i1 , p t-1 i2 +Δ t i2 , …)
[0033] In step S31, the motion vector estimation unit 31 outputs the estimated motion vector to the range estimation unit 32 at the downstream side, and also outputs the estimated current feature point P t i =(p ti1 , p t i2 , ...) is stored in itself and also output to the downstream range estimation unit 32, and then the process proceeds to step S32. Example EX4 in Fig. 4 shows a schematic example of estimating the movement of feature points by estimating motion vectors between times t-1 and t for two object ranges.
[0034] In addition, the motion vector estimation unit 31 estimates the motion of the frame o at the previous time t−1. t-1 Feature point P in t-1 i =(p t-1 i1 , p t-1 i2 , …) is one of the following three cases: <If the current time is t=2, which is the time following the initial time t=1, then (1) below> (1) The feature points obtained by the first feature point detection unit 24 for this initial time t=1 <For each current time t greater than or equal to 3 after this, either (2) or (3) below> (2) If the result of step S4 is negative at the previous past time t-1 and the correction unit 5 is not applied, the feature point obtained by the tracking end determination unit 33 at this past time t-1 (3) If the result of step S4 is a positive determination at the previous past time t-1 and the correction unit 5 is applied, the feature point obtained by the second feature point detection unit 54 at this past time t-1
[0035] <Step S32...Range Estimation Unit 32> In step S32, the range estimation unit 32 estimates the motion vector Δ t i =(Δ t i1 ,Δ t i2 ,…) to find the object range F t = (f t 1 , f t 2, ...) and outputs the estimation result to the tracking end determination unit 33, and then the process proceeds to step S33. (Note that if the determination in step S4 described later is positive and the correction unit 5 is applied, the estimation result is also output to the position correction unit 52, as shown by the line L52 in FIG. 1.)
[0036] Here, the object range F t Each object specified by index i in t i Each element f t i f t i = (x, y, w, h), where x and y are the x and y coordinates of the upper left corner of the rectangle, and w and h are the width and height of the rectangle. t is the object range B obtained in the initialization part 2 t (t=1) can be obtained in the same format, but the final tracking result is the range B t Since is determined through the processing of the tracking end determination unit 33 and the object detection necessity determination unit 4 in the subsequent stage, in order to clearly indicate that it is a range as an intermediate result, the variable notation is distinguished and the range F t (He states that.)
[0037] In the range estimation unit 32, the object range b t i The motion vector Δ t i (The "b t-1 i →b t i The motion vectors of multiple feature points between t i ) to obtain the motion vector Δ t i The change in the object range between times t-1 and t is reflected in b t-1 i →b t i By estimating the change in position and size of the object range during the time t, the object range b ti Estimate.
[0038] Specifically, the range estimation unit 32 calculates Δx t i , Δy t i Δ t i The object range F is calculated by adding the average of the motion vectors as the x and y components of the t (Of the position and size that define the range, the position is updated and estimated from the previous time t-1.) In other words, the object range can be estimated by assuming that the representative positions of multiple feature points for each object are linked to the position of the object range. f t i = f t-1 i + (average(Δx t i ), average(Δy t i ), 0, 0) …Equation (2) Alternatively, a median that is robust to outliers may be used, as shown in the following formula (3). f t i = f t-1 i + (median(Δx t i ), median(Δy t i ), 0, 0) …Equation (3)
[0039] The range estimation unit 32 further updates the object range by estimating the size (w, h) of each object after updating the position (x, y) of each object using the above equations (2) and (3). t-1 Feature point P in t-1 i = (p t-1 i1 , p t-1 i2 , …) and the distribution of frame o t Feature point P in t i= (p t-1 i1 +Δ t i1 , p t-1 i2 +Δ t i2 , …) and the variance ratio α i is calculated as shown in the following formula (4). (The ratio α is used as a ratio of standard deviations, not variance, to quantify the variation in the arrangement of points.) i may be calculated.) α i = var(P t i ) / var(P t-1 i ) …Equation (4)
[0040] And f t i f t-1 i Alpha i The size (w, h) of the object is updated by estimating it as shown in the following formula (5) by accumulating the above. Here, ○ indicates the product of each element of a vector. In other words, the range estimation unit 32 can estimate the size of the object range by assuming that the magnitude of the variation in the positions of multiple feature points for each object is linked to the size of the object range. f t i =f t-1 i ○(1, 1, α i , α i ) …Equation (5)
[0041] Example EX5 in FIG. 4 shows a schematic example in which the position and size of one of the two object ranges between times t-1 and t are updated and estimated using the above method.
[0042] Note that, since noise may occur in feature points estimated by motion vectors, it is desirable for the range estimation unit 32 to remove such feature points before estimating the range, for example, using the Hotelling theory (a method for detecting outliers assuming that data follows a normal distribution), which is an existing method disclosed in the following Non-Patent Document 3. (Note that information on the removal may also be shared with the motion vector estimation unit 31, and the motion vector estimation unit 31 may remove the outlier from the feature points that are the subject of motion estimation at the time t+1 next to the current time t.) [Non-Patent Document 3] Tsuyoshi Ide, "Introduction to Anomaly Detection Using Machine Learning: A Practical Guide Using R", 2015.
[0043] Note that the frame o at the previous time t−1, which is the frame to be updated by the range estimation unit 32, t-1 In range B t-1 As described above with respect to the motion vector estimation unit 31, the following three cases apply. t-1 =B t-1 " as range F t-1 The value of (f t-1 i ∈F t-1 ) and then apply the above formulas (2) to (4). <If the current time is t=2, which is the time following the initial time t=1, then (1) below> (1) The range obtained by the first feature point detection unit 24 for this initial time t=1 <For each current time t greater than or equal to 3 after this, either (2) or (3) below> (2) If the result of step S4 is negative at the previous past time t-1 and the correction unit 5 is not applied, the range obtained by the tracking end determination unit 33 at this past time t-1 (3) If the result of step S4 is a positive determination at the previous past time t-1 and the correction unit 5 is applied, the range obtained by the second feature point detection unit 54 at this past time t-1
[0044] <Step S33...Tracking end determination unit 33> In step S33, the tracking end determination unit 33 determines the object to be tracked and estimates the object range F t The object range F after deleting the object judged to be finished t (and the feature points P t i ) to the object detection necessity determination unit 4, and then the process proceeds to step S4.
[0045] The determination in the tracking end determination unit 33 is, for example, based on the “rectangle image b” from one time before the object range to the current time for objects linked by the same object ID. t-1 i →Rectangular image b t i The normalized cross-correlation (NCC) between the rectangular image b and the target object is calculated, and tracking is terminated when the normalized cross-correlation (NCC) is equal to or smaller than a predetermined threshold. (That is, the higher the similarity, the larger the NCC value.) t-1 i and rectangular image b t i If it is determined that the object range b is dissimilar, the tracking ends. t i If the number of feature points included in the object falls below a predetermined threshold, the object tracking is also determined to be terminated.
[0046] <Step S4...Object detection necessity determination unit 4> In step S4, the object detection necessity determination unit 4 detects the frame o t Regarding the object range F obtained by the tracking end determination unit 33 of the tracking unit 3, it is determined whether object detection is necessary (object detection by the correction unit 5 is necessary separately from object tracking by the tracking unit 3 in the above steps S31, S32, and S33). If it is determined that object detection is necessary, the result is output to the correction unit 5 as shown by the line L5 in FIG. 1, and the process proceeds to step S51. If it is determined that object detection is unnecessary, the tracking end determination unit 33 of the tracking unit 3 outputs the result to the correction unit 5 as shown by the line L5 in FIG. t (and the feature points P t i ) was finally confirmed as the object range B t As (i.e., B t Each element of the individual object range b t i At i =f t i The result is output to the output unit 6, and the process proceeds to step S6.
[0047] Specifically, for example, the object detection necessity determination unit 4 may use a predetermined value N and determine that object detection should be performed when t is a multiple of N. (That is, by periodically performing object detection with a period of N, object tracking by the tracking unit 2 N-1 times and object detection by the correction unit 5 once may be repeated.)
[0048] Alternatively, the object detection necessity determination unit 4 may determine whether or not object detection is necessary for each frame. For example, t =(f t 1 ,f t 2 , …) (total number of objects being tracked) is F t-1 =(f t-1 1 ,f t-1 2 When it is determined that the number of frames has decreased more rapidly than the total number of frames (for example, when it is determined that the number of frames has decreased more rapidly by a threshold value), object detection may be performed for a predetermined number of frames. i If at least one of the variance ratios is equal to or exceeds a predetermined threshold, it is possible to determine that the cluster of feature points is separated into an object other than the tracking target, and perform object detection. i If even one object has feature points that are equal to or smaller than a predetermined threshold value, it may be assumed that the feature points are concentrated on another object, and object detection may be performed.
[0049] That is, the variance ratio α i If it is judged to be large above a certain threshold, or if it is judged to be small below another threshold, the variance ratio α i This corresponds to a case where it is determined that the change in the size of the image is rapid, and when it is determined that the change is rapid, it may be determined that object detection is to be performed.
[0050] <Step S51...Second detection and extraction unit 51> In step S51, the second detection and extraction unit 51 detects frame o at the current time t. t Therefore, the range of each object D t =(d t 1 ,d t 2 , . . . ) and extracts tracking features in each object range, and outputs these detection and extraction results to range correction unit 52, after which the process proceeds to step S52.
[0051] The process of detecting the object range and the process of extracting the tracking features in the second detection / extraction section 51 are similar to the processes in the first detection section 21, and therefore a duplicated description will be omitted.
[0052] <Step S52... Range correction unit 52> In step S52, the range correction unit 52 calculates the frame o t The object range D obtained by the second detection and extraction unit 51 t and the object range F obtained from the range estimation unit 32 based on the output of the line L52 in FIG. t The associated object range is output to the tracking start / end determination unit 53, and the process proceeds to step S53.
[0053] The role of this correspondence is to achieve the following two goals. (1) Object ID in the object tracking result continued up to the current time t (object range F t The object ID in the object detection result newly performed at the current time t (object range D t In the example EX6 of FIG. 4, two object ranges f in the object range Ft are associated with each other, and object tracking is continued after time t by using the same ID. t 1 ,f t 2 There are two object ranges d t 1 ,d t2 When there is "f t 1 and d t 1 " and "f t 2 and d t 2 " is associated with the following example. (2) According to the result of the correspondence, the object range F obtained from the range estimation unit 32 by the output of the line L52 in FIG. t (updating the object regions that continue to exist, adding newly appeared object regions (and setting the positions and sizes of these object regions), and deleting object regions that have disappeared). This correction is executed by the tracking end start determination unit 53 at the downstream side.
[0054] Specifically, the range correction unit 52 corrects the object range D obtained by the second detection and extraction unit 51. t In the set d, which is determined to have a large occlusion between objects (large overlap rate between rectangles), t i -d t j If there is such a set of d t i -d t j Both (d t i and d t j In both cases, the matching is done based on (B) feature points and rectangles. t i is in the range F t =(f t 1 ,f t 2 , …) in the range d t j is in the range F t =(f t 1 ,f t 2 , …) is determined. However, there may be cases where no correspondence can be determined.) On the other hand, for object ranges d t iFor , we perform (A) rectangle-to-rectangle matching. (i.e., for range d t i is in the range F t =(f t 1 ,f t 2 , …) is matched to one of them. Note that there may be cases where no match can be found.) The correspondence between each is as follows:
[0055] (A) Rectangle-to-Rectangle Matching The correspondence can be performed by using a cost based on the rectangular range as the cost of assigning rectangle correspondence, for example, by the Hungarian method, which is an existing method disclosed in the following Non-Patent Document 4. (Note that the correspondence using the Hungarian method may result in no correspondence because no corresponding rectangle exists.) [Non-patent document 4] N. Wojke, A. Bewley, and D. Paulus, "Simple online and realtime tracking with a deep association metric," Proceedings of the 24th IEEE International Conference on Image Processing (ICIP), pp.3645-3649, 2017.
[0056] F t Among the object range rectangles of (object tracking results), D t The object range rectangle of the (new object detection result) is the range b t i A t i = d t i and the object ID z t i Also, D is determined to be the same. t Of these, Hungarian law, etc. tFor those that could not be associated with the object range B, an additional process is performed by using the object range B that has already been obtained as a confirmed result in the past a number of times (a≧1). t-a For example, if a=10, the object range B from 10 frames ago may be tried to be associated with the object range B from 10 frames ago. t-10 = (b t-10 1 , b t-10 2 , …), it may be possible to try to match those that have not yet been matched by referring to the object ID. Similarly, it may try to match the object range B in a predetermined past range by setting a=1, 2, 3, …, A (A is a predetermined upper limit value) etc. until matching is successful. t-k In this case, since the position of the same object may change significantly, instead of performing rectangle-to-rectangle matching using the Hungarian method or the like, matching may be performed using features (feature obtained by the first detection / extraction unit 21 or the second detection / extraction unit 51 at the time in the past closest to the current time t). t i A t i = d t i and confirm it as object ID z t i are also determined to be the same.
[0057] Example EX7 in Figure 5 shows an example where past object detection results are available. t Of the two objects (people) in the frame, the upper one is the object in the previous frame. t-1 In the previous frame, the object is occluded by another object and cannot be matched. t-2 For the bottom object, the correspondence is possible in two frames o t-1 , o t-2 Correspondence is possible in either case.
[0058] (B) Matching based on feature points and rectangles When objects are largely occluded, matching between rectangles often fails. Therefore, we consider pairs dt i -d t j If exists, then F t Feature points P of each object in (object tracking result) t i = (p t i1 , p t i2 , …) and D t Object range d in (new object detection result) t i (and d t j For example, ((p t 11 , p t 12 , …), (p t 21 , p t 22 , …), …) and (d t 1 , d t 2 For all pairs between p t ij ,d t k ) and then matching may be performed on a voting basis only for pairs whose distance is equal to or smaller than a predetermined threshold.
[0059] Note that the above distance dist(p t ij ,d t k ) object range d t k The position of may be the upper left coordinate (x, y) of the rectangle, or the center position (x+w / 2, y+h / 2) of the rectangle taking into account the size (w, h), or any other predetermined position within the rectangle. t ij is obtained by additionally sampling from any of the joint points or its surroundings, the position of the corresponding joint point (which can be detected by the second feature point detection unit 54 at the subsequent stage, but which is obtained in advance) may be used.
[0060] In the example EX8 in Figure 5, an example of such voting-based matching is shown. t 1 -d t 2 In the situation where the first person is present, the first object (three feature points P t 1 =(p t 11 , p t 12 , p t 13 ) has p t 11 , p t 12 By object range d t 1 The second object has three feature points P t 2 =(p t 21 , p t 22 , p t 23 ) has p t 21 , p t 22 By object range d t 2 The results of these two votes give us the following matching result: Feature point P t 1 The object tracking range f associated with t 1 is the new object detection range d t 1 Corresponds to. Feature point P t 2 The object tracking range f associated with t 2 is the new object detection range d t 2 Corresponds to.
[0061] <Step S53...Tracking end / start determination unit 53> In step S53, the tracking end / start determination unit 53 determines the result obtained by the range correction unit 52 (area Ft and area D t Based on the result of the matching, the object to be tracked from the current time t onwards is determined, and the object to be tracked from the current time t onwards is determined. t Then, this object area B t to the second feature point detection unit 54, and then the process proceeds to step S54.
[0062] That is, in the tracking end start determination unit 53, (1) D t = (d t 1 , d t 2 , …), the object that could not be matched is treated as a new object that has appeared at the current time t and tracking is started. (2) On the other hand, F t = (f t 1 , f t 2 , …), those that could not be matched are treated as objects that have disappeared at the current time t and tracking is terminated. (3) Region F t and area D t If an object is associated with another object, the object tracking is continued even at the current time t. Then, the object area B that is the object tracking target after the current time t is determined. t This confirms the entirety of the above.
[0063] Example EX9 in Figure 5 shows a schematic example of the judgment process of the tracking end start judgment unit 53, in which one object is judged to have existed at the previous time t-1 and then disappeared at the current time t, one object is judged to have newly appeared at the current time t, and one object is judged to have continued to exist at times t-1, t, and t+1.
[0064] <Step S54...Second feature point detection unit 54> In step S54, the frame o at the current time t obtained by the tracking end start determination unit 53 is t Object range B that is the target of object tracking in t =(b t1 , b t 2 , …)(t=1) for each b t i (i=1,2,…) to the characteristic point P t i =(p t i1 , p t i2 , …)(t=1) and detect this feature point P t i and object range B t to the output unit 6 and the tracking unit 3, and then the process proceeds to step S6.
[0065] The processing of the second feature point detection unit 54 is the same as that of the first feature point detection unit 24, so a duplicated description will be omitted. t i to the motion vector estimation unit 31, and the object range B t to the range estimation unit 32, thereby enabling the tracking unit 3 to track the object at a later time t+1 or later.
[0066] As described above, the processing of the correction unit 5 (steps S51, S52, S53, S54) is equivalent to the processing of the initialization unit 2 (steps S21, S23, S24) with the addition of a process of finding a correspondence with the object tracking results in the tracking unit 3 and making them consistent.
[0067] <Step S6... Output section 6> In step S6, the output unit 6 outputs the frame o t Object tracking result by the object tracking device 10 in the object range B t and feature point P t i ) and then the process proceeds to step S61. The output unit 6 is configured as a display in terms of hardware, and it is sufficient to output the object tracking result by displaying it to the user. t For object range B t and feature point P ti may be displayed superimposed on the corresponding position.
[0068] As is clear from the above description, the object tracking result (object range B t and feature point P t i ) is the result obtained by the initialization unit 2 if steps S21, S23, and S24 are followed for the current time t, is the result obtained by the tracking unit 3 if a negative judgment is obtained from steps S31, S32, and S33 and then step S4 (the result being further output from the object detection necessity judgment unit 4), and is the result obtained by the correction unit 5 if a positive judgment is obtained in step S4 and then steps S51, S52, S53, and S54 are followed.
[0069] <Step S61...Time Update> In step S61, the current time t is updated to the next current time t+1, and then the process returns to step S1, whereby the processing of the object tracking device 10 is repeated for the next time t+1 in the same manner as described above.
[0070] As described above, according to the object tracking device 10 of the embodiment of the present invention, since the tracking unit 3 basically tracks only the feature points around the position where there is little change, tracking is possible even in a situation where non-detection is likely to occur, such as occlusion, in terms of accuracy, and since the motion vector only needs to be calculated around the feature points in terms of speed, high-speed processing is possible. Furthermore, since the tracking result of the whole body range of a person, etc. can be obtained through a rectangle while tracking only the feature points to realize high-speed calculation, various applications such as behavior recognition can be secured for the tracking of a person, etc. by the object tracking device 10. Furthermore, since the object detection result by the correction unit 5 is intermittently reflected in the object tracking result on the video according to the judgment of the object detection necessity judgment unit 4, it is possible to continue object tracking with high accuracy even when a new object appears in the middle of the video.
[0071] The example in FIG. 6 is a schematic example of the process in which the object tracking device 10 tracks an object according to the flow in FIG. 2, and shows how the object tracking device 10 performs high-speed object tracking using motion vectors for each of frames o1 to o6 on the video at times t=1 to 6, while intermittently correcting the object tracking results by object detection at appropriate times.
[0072] Various supplementary, additional, and alternative examples will be described below.
[0073] (1) According to an embodiment of the present invention, as an application example, it is possible to automatically monitor the movement status of multiple (or a large number of) people in a remote location in real time for a video image capturing the situation in which the people are present. This makes it possible to monitor the situation of people in a remote location in detail and to provide advice to the remote location based on the monitoring results, without necessarily requiring the user to actually travel to the remote location. Therefore, it is possible to reduce carbon dioxide emissions by saving the energy resources required for user movement, which makes it possible to contribute to Goal 13 of the United Nations-led Sustainable Development Goals (SDGs), "Take urgent action to combat climate change and its impacts."
[0074] (2) According to the flow of FIG. 2, it is possible to perform real-time object tracking on frame images captured by the imaging unit 1 at each real-time time t=1, 2, .... However, it is also possible to apply the flow of FIG. 2 to images that have been captured in advance and saved as footage spanning a certain period of time.
[0075] (3) FIG. 7 is a diagram showing an example of a hardware configuration in a general computer device 70. The object tracking device 10 can be realized as one or more computer devices 70 having such a configuration. When the object tracking device 10 is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes part or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, a keyboard, a touch panel, or the like, a camera 78, and a bus BS for transmitting and receiving data between them.
[0076] Each functional unit of the object tracking device 10 can be realized by a CPU 71 and / or a GPU 72 that reads a predetermined program corresponding to the function of each unit from a ROM 74 and executes it. Both the CPU 71 and the GPU 72 are a type of computing device (processor). Here, when display-related processing is performed, a display 76 also operates in conjunction with the GPU 72, and when communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the GPU 72. Each camera of the photographing unit 1 may be realized as a camera 78. [Explanation of symbols]
[0077] 10: object tracking device, 1: imaging section, 2: initialization section, 3: tracking section, 4: object detection necessity determination section, 5: correction section, 6: output section
Claims
1. a tracking unit that tracks an object region, which is a region of an object captured in a frame, continuously between frames captured at different times in a video; a correction unit that corrects an object region tracked by the tracking unit for a portion of frames in the video and causes the tracking unit to continue tracking the corrected object region, the correction unit detects objects from the portion of frames, and performs the correction by associating each of the detected object regions with each of the object regions tracked by the tracking unit; The tracking unit tracks feature points of an object to track the object region; the correction unit detects objects from the portion of frames, and determines whether or not an overlap rate between each of the detected object regions and other object regions is equal to or greater than a certain rate; For object regions determined to be equal to or greater than the certain percentage, a correspondence is established between each of the detected object regions and each of the object regions tracked by the tracking unit by associating feature points in the object regions with feature points in the object regions tracked by the tracking unit that are determined to have small distances therebetween; An object tracking device characterized by the fact that, for object areas determined to be less than a certain percentage, a correspondence is established between each of the detected object areas and each of the object areas tracked by the tracking unit based on the relative positions of the areas.
2. a tracking unit that tracks an object region, which is a region of an object captured in a frame, continuously between frames captured at different times in a video; a correction unit that corrects an object region tracked by the tracking unit for a portion of frames in the video and causes the tracking unit to continue tracking the corrected object region, the correction unit detects objects from the portion of frames, and performs the correction by associating each of the detected object regions with each of the object regions tracked by the tracking unit; The tracking unit tracks feature points of an object to track the object region; the tracking unit tracks a plurality of feature points for each object region, and tracks a size of the object region as having a size that is linked to a degree of variation in positions of the plurality of feature points; The object tracking device further comprises an object detection necessity determination unit that, when a change in size of at least one of the tracked object regions is determined to be sudden by threshold judgment, determines that the frame in the video in which the sudden change occurs corresponds to the portion of frames and is therefore subject to correction by the correction unit.
3. a tracking unit that tracks an object region, which is a region of an object captured in a frame, continuously between frames captured at different times in a video; a correction unit that corrects an object region tracked by the tracking unit for a portion of frames in the video and causes the tracking unit to continue tracking the corrected object region, the correction unit detects objects from the portion of frames, and performs the correction by associating each of the detected object regions with each of the object regions tracked by the tracking unit; When it is determined that the number of object regions tracked by the tracking unit has significantly decreased between the previous time and the current time by threshold value determination, The object tracking device further comprises an object detection unit necessity determination unit that determines that the frame at the current time corresponds to the portion of frames and is therefore a target for correction by the correction unit.
4. 4. The object tracking device according to claim 1, wherein the tracking unit tracks the object region by tracking feature points of the object.
5. 5. The object tracking device according to claim 4, wherein the tracking unit tracks the object region by tracking a plurality of feature points for each object region, tracking the position of the object region as having a position linked to representative positions of the plurality of feature points, and tracking the size of the object region as having a size linked to a magnitude of variation in the positions of the plurality of feature points.
6. each of said objects being an individual person; the feature points of the object tracked by the tracking unit include joint points, such as head joint points and shoulder joint points, that are located at a height equal to or higher than the chest of a person, and do not include points located lower than the chest; 6. The object tracking device according to claim 4, wherein the object region tracked by the tracking unit is a region covering the entire body of a person.
7. each of said objects being an individual person; the feature points of the object tracked by the tracking unit include only predetermined types of joint points that are expected to have little change in positional relationship between joint points in a person, such as a head, shoulders, torso, or buttocks; 6. The object tracking device according to claim 4, wherein the object region tracked by the tracking unit is a region covering the entire body of a person.
8. 8. The object tracking device according to claim 6, wherein the feature points of the object tracked by the tracking unit include points sampled from around the joint points.
9. 9. The object tracking device according to claim 4, wherein the tracking unit tracks feature points of the object by estimating a motion vector between frames.
10. The object tracking device according to claim 9, characterized in that the tracking unit excludes from the tracking target any feature point for which the motion vector could not be estimated, and excludes from the tracking target any object region in which the number of tracked feature points is less than a threshold value.
11. The object tracking device according to any one of claims 1 to 10, characterized in that the tracking unit excludes from the tracking target an object region being tracked that is determined to have a low similarity between the object region at the immediately preceding time and the object region at the current time.
12. the correction unit performs correspondence between each of the detected object regions and each of the object regions tracked by the tracking unit; If there is an object region that cannot be associated with the object region tracked by the tracking unit, the object in the object region is deemed to have disappeared and is excluded from targets to be continuously tracked by the tracking unit; The object tracking device according to any one of claims 1 to 11, characterized in that if there is an object region among the detected object regions that cannot be associated, the object in the object region is added to the list of objects to be continued to be tracked by the tracking unit as a newly appeared object.
13. An initialization unit that detects an object from a frame at an initial time of the video, 13. The object tracking device according to claim 1, wherein the tracking unit performs object tracking for each of the object regions detected in the frame at the initial time, targeting frames after the initial time.
14. A program for causing a computer to function as the object tracking device according to any one of claims 1 to 13.
Citation Information
Patent Citations
Image characteristic tracing device and image characteristic tracing method
JP1997214945A
Hand shake correcting method, program for hand shake correcting method, recording medium where program for hand shake correcting method is recorded, and hand shake correcting device
JP2008005109A
Object tracking device and program
JP2010039617A
Object tracking device, object tracking system, object tracking method, display control device, object detection device, program, and recording medium
JP2016071830A
Tracking device
JP2017076181A
