Object recognition and tracking method and device

Abstract

The present disclosure provides an object recognition and tracking method and device. The object recognition and tracking method includes detecting M first objects included in a target frame image; obtaining N second objects included in tracking data in the first video data, and the matching weight of each second object and the first object changes with the time interval between the image where the current second object is located and the target frame image; matching the M first objects with the N second objects to determine the corresponding relationship between each first object and each second object; and tracking the first object according to the matching results of the M first objects and the N second objects. In this way, by determining the number of image frames used for tracking data according to the matching weights of the second objects and the first objects, it is helpful to reduce the number of image frames of the first video data required to be used, and improve the efficiency of object recognition and tracking.

Description

Technical Field

[0001] The present disclosure relates to the technical field of object tracking, and in particular to an object recognition and tracking method and device. Background Art

[0002] In the process of video monitoring and video data processing, multi-target tracking is a common and practical technology. The inventors of the present disclosure found that the target can be tracked by first identifying the target and then accumulating multiple frames, but this method requires a high amount of calculation. When the number of accumulated frames is large, there is a large calculation delay, which reduces the tracking efficiency. Summary of the invention

[0003] The embodiments of the present disclosure provide an object recognition and tracking method and device to improve the efficiency of object recognition and tracking.

[0004] In a first aspect, an embodiment of the present disclosure provides an object recognition and tracking method, comprising the following steps:

[0005] Detecting M first objects included in the target frame image, where M is a positive integer;

[0006] Acquire N second objects included in the tracking data in the first video data, wherein the first video data includes L frame images before the target frame image, wherein N and L are both positive integers, and L is determined according to the matching weights of the second objects and the first object;

[0007] Matching the M first objects with the N second objects to determine a corresponding relationship between each first object and each second object, wherein a matching weight between each second object and the first object varies with a time interval between an image where a current second object is located and the target frame image;

[0008] The first object is tracked according to a matching result between the M first objects and the N second objects.

[0009] In some embodiments, matching the M first objects with the N second objects to determine the corresponding relationship between the first objects and the second objects includes:

[0010] respectively calculating feature distances between feature vectors of the N second objects in each of the L frames of images and feature vectors of the M first objects;

[0011] sequentially calculating a weighted moving average of the characteristic distances between each second object and the first object, wherein the matching weight of each second object and the first object increases as the time interval between the current image where the second object is located and the target frame image decreases;

[0012] The first object and the second object whose weighted moving average value is greater than a preset threshold and whose value is the largest correspond to each other.

[0013] In some embodiments, the matching weight of the second object and the first object changes exponentially as the time interval between the image in which the second object resides and the target frame image decreases.

[0014] In some embodiments, the detecting the M first objects included in the target frame image includes:

[0015] Extracting feature vectors of M first objects included in the target frame image respectively;

[0016] The obtaining of N second objects included in the tracking data in the first video data comprises:

[0017] generating a feature vector corresponding to each tracked second object in the first video data;

[0018] The matching the M first objects with the N second objects includes:

[0019] Calculating a feature distance between a feature vector of each first object and a feature vector of each second object;

[0020] A maximum match between a feature vector of each of the first objects and feature vectors of the N second objects is determined according to the feature distance, and the maximum match is used as the corresponding first object and second object.

[0021] In some embodiments, tracking the first object according to the matching results between the M first objects and the N second objects includes:

[0022] If there is a first object matching the second object, tracking the corresponding first object and the second object as the same object;

[0023] If there is no first object matching the second object, a new tracking object is added according to the second object.

[0024] In a second aspect, an embodiment of the present disclosure provides an object recognition and tracking device, including:

[0025] A detection module, configured to detect M first objects included in the target frame image, where M is a positive integer;

[0026] A tracking data acquisition module, used to acquire N second objects included in the tracking data in the first video data, wherein the first video data includes L frame images before the target frame image, wherein N and L are both positive integers, and L is determined according to the matching weight of the second objects and the first object;

[0027] a matching module, configured to match the M first objects with the N second objects to determine a corresponding relationship between each first object and each second object, wherein a matching weight between each second object and the first object varies with a time interval between an image where a current second object is located and the target frame image;

[0028] A tracking module is used to track the first object according to the matching result between the M first objects and the N second objects.

[0029] In some embodiments, the matching module includes:

[0030] A distance calculation submodule, used for respectively calculating the characteristic distances between the characteristic vectors of the N second objects in each frame of the L frames and the characteristic vectors of the M first objects;

[0031] A weighted moving average calculation submodule, used to sequentially calculate the weighted moving average of the characteristic distance between each second object and the first object, wherein the matching weight of each second object and the first object increases as the time interval between the current image of the second object and the target frame image decreases;

[0032] The matching submodule is used to match the first object and the second object whose weighted moving average value is greater than a preset threshold and is the largest.

[0033] In some embodiments, the matching weight of the second object and the first object changes exponentially as the time interval between the image in which the second object resides and the target frame image decreases.

[0034] In some embodiments, the detection module is specifically used to respectively extract feature vectors of M first objects included in the target frame image;

[0035] The tracking data acquisition module is specifically used to generate a feature vector corresponding to each tracked second object in the first video data;

[0036] The matching module comprises:

[0037] The distance calculation submodule is specifically used to calculate the characteristic distance between the characteristic vector of each first object and the characteristic vectors of the N second objects;

[0038] The matching submodule is specifically configured to determine the maximum match between the feature vector of each first object and the feature vector of each second object according to the feature distance, and use the maximum match as the corresponding first object and second object.

[0039] In some embodiments, the tracking module is specifically used to:

[0040] If there is a first object matching the second object, tracking the corresponding first object and the second object as the same object;

[0041] If there is no first object matching the second object, a new tracking object is added according to the second object.

[0042] The disclosed embodiment detects M first objects included in a target frame image, where M is a positive integer; obtains N second objects included in tracking data in the first video data, where the first video data includes L frame images before the target frame image, where N and L are both positive integers, and L is determined based on the matching weight between the second object and the first object; matches the M first objects with the N second objects to determine the corresponding relationship between the first object and the second object; and tracks the first object based on the matching results between the M first objects and the N second objects. In this way, by determining the number of image frames used for tracking data based on the matching weight between the second object and the first object, it helps to reduce the number of image frames of the first video data required to be used, and improve the efficiency of object recognition and tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the description of the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 is a flow chart of an object recognition and tracking method provided by an embodiment of the present disclosure;

[0045] Figure 2 It is a structural diagram of an object recognition and tracking device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0047] An embodiment of the present disclosure provides an object recognition and tracking method.

[0048] like Figure 1 As shown, in one embodiment, the method includes the following steps:

[0049] Step 101: Detect M first objects included in a target frame image, where M is a positive integer.

[0050] In the embodiment of the present disclosure, M first objects included in the target frame image are first detected, and the first objects may be people or other objects.

[0051] In one embodiment, step 101 further includes:

[0052] Feature vectors of M first objects included in the target frame image are respectively extracted.

[0053] During implementation, a detection frame corresponding to each identified first object may be first established, and a feature vector corresponding to the detection frame may be further generated as a feature vector of the corresponding first object. During implementation, the feature vector of each first object may be extracted by a feature extraction algorithm including but not limited to deepsort.

[0054] Step 102: Acquire N second objects included in the tracking data in the first video data, where the first video data includes L frame images before the target frame image, and both N and L are positive integers.

[0055] Object recognition is mainly aimed at the target frame image, while object tracking is mainly to match the image recognized in the historical video data with the object in the target frame image. In this way, the object's motion trajectory, motion speed and other related parameters can be obtained to achieve object tracking.

[0056] In the technical solution of this embodiment, the first video data refers to the above-mentioned historical video data, and the target frame refers to a frame of image that is located after the first video data and for which continuous tracking of the object included therein has not yet been achieved.

[0057] Furthermore, in one embodiment, step 102 includes:

[0058] A feature vector corresponding to each tracked second object in the first video data is generated.

[0059] Similar to the above step 101, in this embodiment, a tracking frame corresponding to each tracked second object is firstly established, and a feature vector corresponding to each tracking frame is further generated.

[0060] It should be understood that the format of the feature vector of the tracking frame and the feature vector of the detection frame is the same. For example, if the feature vector of the detection frame is a 128-dimensional feature vector, then the feature vector of the tracking frame is also a 128-dimensional feature vector.

[0061] Step 103: Match the M first objects with the N second objects to determine a corresponding relationship between each first object and each second object.

[0062] After determining the M first objects in the target frame image and the N second objects in the tracking data, the first objects are further matched with the second objects, thereby determining the corresponding relationship between the first objects and the second objects.

[0063] During implementation, the M first objects and the N second objects are matched one by one. For example, the first first object is matched with the first to Nth second objects in sequence, and a total of N groups of matching data are obtained. Next, the second first object is matched with the first to Nth second objects in sequence, and N groups of matching data are also obtained. This can be deduced by analogy, and a total of M*N groups of matching data can be obtained, that is, the correspondence between each first object and each second object.

[0064] The matching weight of each second object and the first object varies with the time interval between the image where the current second object is located and the target frame image. It can be understood that the image of each second object exists in multiple frames of images in the L frame image included in the first video data. In order to obtain tracking data for the second object, the data of the second object in the multiple frames of images in the L frame image will be introduced.

[0065] On the scale of time change, the data of each second object in different frame images has different degrees of influence on the data of the second object in the target frame. Therefore, in this embodiment, a matching weight that changes with time is further set to improve the tracking effect for the second object.

[0066] In one embodiment, step 103 includes:

[0067] Calculating a feature distance between a feature vector of each of the first objects and feature vectors of the N second objects;

[0068] A maximum match between a feature vector of each first object and a feature vector of each second object is determined according to the feature distance, and the maximum match is used as the corresponding first object and second object.

[0069] In the technical solution of this embodiment, the characteristic distance between the characteristic vectors of the first object and the second object can be calculated by cosine distance, or by other methods such as Euclidean distance.

[0070] Take the cosine distance as an example.

[0071] Where dist is the feature distance, x and y are the feature vectors of the first object and the second object respectively.

[0072] Furthermore, after the feature distance between the feature vectors of the first object and the second object is calculated, the maximum match between the first object and the second object may be determined using the Hungarian algorithm.

[0073] The maximum match of two sets refers to the match result with the most matching results of objects in the two sets. Therefore, the maximum match of the set of first objects and the set of second objects is actually to determine the most likely one-to-one correspondence between the first object and the second object. That is to say, the maximum match result can be considered as a one-to-one correspondence between the M first objects included in the target frame image and the N second objects included in the tracking data.

[0074] The Hungarian algorithm is used to find augmenting paths, and the augmenting paths are used to find the maximum matching of the bipartite graph.

[0075] A bipartite graph is defined as follows: if the node set V of the graph G can be divided into two non-empty subsets V1 and V2, and the two nodes m and n associated with any edge mn of the graph G belong to these two subsets respectively, then G is a bipartite graph.

[0076] The basic steps of the Hungarian algorithm can be summarized as follows:

[0077] (1) Find the object M that can be matched by the current node m. If the object M has been matched, go to step 3; otherwise, go to step 2.

[0078] (2) Record the matching object of object M as the current object m, and go to step 6;

[0079] (3) Find the object n that the object M has matched, and find out whether n can match another object. If so, go to step 4; otherwise, go to step 5.

[0080] (4) Update the matching object n to another object N, update the matching object of object M to aa, and go to step 6;

[0081] (5) Node m searches for the next matching object. If it exists, it proceeds to step 1. Otherwise, it means that the current node m has no matching object, so it proceeds to step 6.

[0082] (6) Go to the next node and then go to step 1 until everything is completed.

[0083] In the present invention, all first objects are set V1, and all second objects are set V2. Furthermore, the first objects and the second objects are matched according to the feature distance between the feature vectors of the first objects and the second objects.

[0084] Furthermore, the step 103 specifically includes:

[0085] respectively calculating feature distances between feature vectors of the N second objects in each of the L frames of images and feature vectors of the M first objects;

[0086] sequentially calculating a weighted moving average of the characteristic distances between each second object and the first object, wherein the matching weight of each second object and the first object increases as the time interval between the current image where the second object is located and the target frame image decreases;

[0087] The first object and the second object whose weighted moving average value is greater than a preset threshold and whose value is the largest correspond to each other.

[0088] It should be understood that, for the L frame images included in the first video data, the tracking of the second objects therein has been completed, and therefore, the position of each second object in the L frame images included in the first video data can be understood to be known.

[0089] Furthermore, according to the feature vectors of each second object in different frame images in the L frame images, the feature distances between the feature vectors of each object are calculated respectively, and further, the weighted moving average values ​​corresponding to each second object are calculated.

[0090] During implementation, the order of the above steps may also be adjusted, for example, first calculating the weighted moving average of the characteristic vector of each second object, and then calculating the characteristic distance between the weighted moving average of the characteristic vector and the characteristic vector of the first object.

[0091] v t =βv t-1 +(1-β)θ t ……(1)

[0092] In implementation, the weighted moving average can be calculated by the above formula (1), where θ t is the value to be calculated. Taking the weighted moving average of the feature vector of the second target object as an example, then θ t is the feature vector of the target second object at time t. In this embodiment, it can be specifically the feature vector of the t-th frame image in the above L frame images. When different calculation methods are used, θ t It may also be a feature distance between a feature vector of the second object in the t-th frame image of the L-frame images included in the first video data and a feature vector of the first object, which is specifically determined according to the selected calculation method.

[0093] The coefficient β represents the rate of weighted decline, the smaller the value, the faster the decline; v t is the value of the weighted moving average at time t.

[0094] In a specific embodiment, the first objects A1 and A2 are identified in the target frame, and the first video data includes the second objects B1, B2, and B3. The first video includes L frames of images, wherein the feature vector of the second object B2 in each frame of the image is recorded as B 11 , B 12 ……、B 1L-1 , B 1L , and so on.

[0095] During implementation, the characteristic distance between each first object and each second object can be calculated respectively. For example, the characteristic distance between the characteristic vector of the second object B2 in the first frame image in the L frame image and the characteristic vector of the first object A1 is recorded as A1B 11 , and so on, calculate A1B 12 、A1B 13 , ...A1B 1L-1 、A1B 1L .

[0096] Next, calculate A1B 11 、A1B 12 , ...A1B 1L-1 、A1B 1L The weighted moving average of A1B 11 、A1B 12 , ...A1B 1L-1 、A1B 1L As θ1, θ2, ... θ 1L-1 ,θ L Substitute into the above formula (1) to calculate the weighted moving average v t , and the v t A1B 1X , where the matching weight of each second object with the first object increases as the time interval between the current image where the second object is located and the target frame image decreases, that is, A1B 12 、A1B 12 , ...A1B 1L-1 、A1B 1L The weighted weights increase in sequence.

[0097] It can also be understood that, in the L frame images, the closer the image is to the target frame, the greater the correlation between the image and the target frame image is, and the greater the corresponding weight is.

[0098] In another specific implementation, the feature vector B of the second object B2 in each frame image is obtained. 11 , B 12 ……、B 1L-1 , B 1LAfterwards, they are respectively referred to as θ1, θ2, ... 1L-1 ,θ L , and calculate the weighted moving average of these feature vectors using the above formula (1), denoted as B 1x , further calculation B 1x The characteristic distance from the characteristic vector of the first object A1 is recorded as A1B 1X .

[0099] In one embodiment, the weight changes exponentially as the time interval between the current image and the target frame image decreases.

[0100] Furthermore, we can obtain the formula:

[0101] v t =(1-β)(θ t +βθ t-1 +…+β t-1 θ1)……(2)

[0102] Formula (2) is used to calculate the weighted moving average. In this way, the influence of time on the object motion information can be further reflected, and the information in the image with a small time interval with the target frame can be more fully utilized.

[0103] It should be understood that L in this embodiment is determined according to the matching weight between the second object and the first object.

[0104] During implementation, a critical value may be set, for example, 1 / e, where e is a natural base number, approximately equal to 2.71828.... Further, in this embodiment, a weighted decreasing rate is set, that is, β in the above formula (2). For example, when β=0.9, 0.9 10 is approximately equal to 1 / e, so it is considered to be the weighted average of the last 10 values, and accordingly, L is equal to 10; when β = 0.98, 0.9 50 It is approximately equal to 1 / e, so it is considered to be the weighted average of nearly 50 values, and the corresponding L is equal to 50.

[0105] In this way, setting the value of L according to the matching weight of the second object and the first object can not only select valid data and improve the tracking effect of the object, but also help reduce the amount of calculation and improve the tracking efficiency of the object.

[0106] Step 104: Track the first object according to the matching result between the M first objects and the N second objects.

[0107] Specifically, in one embodiment, step 104 includes:

[0108] If there is a first object matching the second object, tracking the corresponding first object and the second object as the same object;

[0109] If there is no first object matching the second object, a new tracking object is added according to the second object.

[0110] In this embodiment, if there is a second object that matches the first object, it means that the first object is likely to be the same object as the second object that matches it. In this way, they are treated as the same object to achieve tracking of the first object. If there is no second object that matches the first object, it means that the first object is likely to have not appeared in the video image before. In this way, the first object is tracked as a new object, thereby continuously detecting and tracking objects in the video image and achieving continuous monitoring of objects in the area corresponding to the video image.

[0111] The disclosed embodiment determines the number of image frames used for tracking data according to the matching weight of the second object and the first object, thereby helping to reduce the number of image frames of the first video data required and improving the efficiency of object recognition and tracking.

[0112] The present disclosure also provides an object recognition and tracking device.

[0113] like Figure 2 As shown, in one embodiment, the object recognition and tracking device 200 includes:

[0114] A detection module 201 is used to detect M first objects included in a target frame image, where M is a positive integer;

[0115] A tracking data acquisition module 202 is used to acquire N second objects included in the tracking data in the first video data, wherein the first video data includes L frame images before the target frame image, wherein N and L are both positive integers, and L is determined according to the matching weight of the second objects and the first object;

[0116] A matching module 203 is used to match the M first objects with the N second objects to determine the corresponding relationship between each first object and each second object, and the matching weight of each second object and the first object varies with the time interval between the image where the current second object is located and the target frame image;

[0117] The tracking module 204 is configured to track the first object according to the matching result between the M first objects and the N second objects.

[0118] In some embodiments, the matching module 203 includes:

[0119] A distance calculation submodule, used for respectively calculating the characteristic distances between the characteristic vectors of the N second objects in each frame of the L frames and the characteristic vectors of the M first objects;

[0120] A weighted moving average calculation submodule, used to sequentially calculate the weighted moving average of the characteristic distance between each second object and the first object, wherein the matching weight of each second object and the first object increases as the time interval between the current image of the second object and the target frame image decreases;

[0121] The matching submodule is used to match the first object and the second object whose weighted moving average value is greater than a preset threshold and is the largest.

[0122] In some embodiments, the matching weight of the second object and the first object changes exponentially as the time interval between the image in which the second object resides and the target frame image decreases.

[0123] In some embodiments, the detection module 201 is specifically used to extract feature vectors of M first objects included in the target frame image respectively;

[0124] The tracking data acquisition module is specifically used to generate a feature vector corresponding to each tracked second object in the first video data;

[0125] The matching module 203 includes:

[0126] The distance calculation submodule is specifically used to calculate the characteristic distance between the characteristic vector of each first object and the characteristic vectors of the N second objects;

[0127] The matching submodule is specifically configured to determine the maximum match between the feature vector of each first object and the feature vector of each second object according to the feature distance, and use the maximum match as the corresponding first object and second object.

[0128] In some embodiments, the tracking module 204 is specifically configured to:

[0129] If there is a first object matching the second object, tracking the corresponding first object and the second object as the same object;

[0130] If there is no first object matching the second object, a new tracking object is added according to the second object.

[0131] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An object recognition and tracking method, comprising the following steps: Detecting M first objects included in the target frame image, where M is a positive integer; Acquire N second objects included in the tracking data in the first video data, wherein the first video data includes L frame images before the target frame image, wherein N and L are both positive integers, and L is determined according to the matching weights of the second objects and the first object; Matching the M first objects with the N second objects to determine a corresponding relationship between each first object and each second object, wherein a matching weight between each second object and the first object varies with a time interval between an image where a current second object is located and the target frame image; Tracking the first object according to a matching result between the M first objects and the N second objects; The matching the M first objects with the N second objects to determine the correspondence between the first objects and the second objects includes: respectively calculating feature distances between feature vectors of the N second objects in each of the L frames of images and feature vectors of the M first objects; sequentially calculating a weighted moving average of the characteristic distances between each second object and the first object, wherein the matching weight of each second object and the first object increases as the time interval between the current image where the second object is located and the target frame image decreases; The first object whose weighted moving average value is greater than a preset threshold and the largest one corresponds to the second object; The matching weight between the second object and the first object changes exponentially as the time interval between the image in which the second object resides and the target frame image decreases.

2. The method according to claim 1, wherein: The detecting of the M first objects included in the target frame image comprises: Extracting feature vectors of M first objects included in the target frame image respectively; The obtaining of N second objects included in the tracking data in the first video data comprises: generating a feature vector corresponding to each tracked second object in the first video data; The matching the M first objects with the N second objects includes: Calculating a feature distance between a feature vector of each first object and a feature vector of each second object; A maximum match between a feature vector of each of the first objects and feature vectors of the N second objects is determined according to the feature distance, and the maximum match is used as the corresponding first object and second object.

3. The method according to claim 1, wherein: Tracking the first object according to the matching result between the M first objects and the N second objects includes: If there is a first object matching the second object, tracking the corresponding first object and the second object as the same object; If there is no first object matching the second object, a new tracking object is added according to the second object.

4. An object recognition and tracking device, comprising: A detection module, configured to detect M first objects included in the target frame image, where M is a positive integer; A tracking data acquisition module, used to acquire N second objects included in the tracking data in the first video data, wherein the first video data includes L frame images before the target frame image, wherein N and L are both positive integers, and L is determined according to the matching weight of the second objects and the first object; a matching module, configured to match the M first objects with the N second objects to determine a corresponding relationship between each first object and each second object, wherein a matching weight between each second object and the first object varies with a time interval between an image where a current second object is located and the target frame image; A tracking module, configured to track the first object according to a matching result between the M first objects and the N second objects; The matching module comprises: A distance calculation submodule, used for respectively calculating the characteristic distances between the characteristic vectors of the N second objects in each frame of the L frames and the characteristic vectors of the M first objects; A weighted moving average calculation submodule, used to sequentially calculate the weighted moving average of the characteristic distance between each second object and the first object, wherein the matching weight of each second object and the first object increases as the time interval between the current image of the second object and the target frame image decreases; A matching submodule, used for matching a first object whose weighted moving average value is greater than a preset threshold and is the largest with a second object; The matching weight between the second object and the first object changes exponentially as the time interval between the image in which the second object resides and the target frame image decreases.

5. The device according to claim 4, wherein: The detection module is specifically used to respectively extract feature vectors of M first objects included in the target frame image; The tracking data acquisition module is specifically used to generate a feature vector corresponding to each tracked second object in the first video data; The matching module comprises: The distance calculation submodule is specifically used to calculate the characteristic distance between the characteristic vector of each first object and the characteristic vectors of the N second objects; The matching submodule is specifically configured to determine the maximum match between the feature vector of each first object and the feature vector of each second object according to the feature distance, and use the maximum match as the corresponding first object and second object.

6. The device according to claim 4, wherein: The tracking module is specifically used for: If there is a first object matching the second object, tracking the corresponding first object and the second object as the same object; If there is no first object matching the second object, a new tracking object is added according to the second object.

Images