Apparatus and method for using neural networks to correlate object detections between frames

Abstract

Apparatuses and methods are disclosed that use neural networks to correlate object detections between frames. A convolutional neural network determines a reference set of two or more sets of feature vectors associated with identified regions corresponding to object detections in a first frame and a candidate set of feature vectors associated with a respective one of the identified regions corresponding to object detections in a second frame. A closest set of feature vectors is determined as a predetermined number of feature vectors of the candidate set that have a smallest distance to each feature vector of the reference set in a feature vector space. For each candidate set, a closeness measure to the reference set of feature vectors is determined as a proportion of the closest set of feature vectors belonging to that candidate set. For each object detection in the second frame, a respective weight is determined based on the determined closeness measures to the reference set of feature vectors of each candidate set corresponding to that object detection. The object detection in the first frame is correlated with one of the object detections in the second frame based on the assigned weights.

Description

Technical Field

[0001] This invention relates to tracking objects in video, and more particularly to using a convolutional neural network to correlate object detection in a first frame with object detection in a second frame. Background Technology

[0002] When tracking multiple objects in a video, a Kalman filter or similar method can be used. In such methods, tracking of multiple objects is updated sequentially frame by frame based on object detections in the frames. To update the tracking, object detections in the current frame are first associated with existing tracking in previous frames. Once the association has been performed, the tracking in the previous frames is updated based on the positions of the associated object detections in the current frame.

[0003] The association can be based on probabilities representing the likelihood that an existing tracked object in a previous frame and an object detected in the current frame are the same object. These probabilities can be based on Kalman filter state variables. For example, Kalman filter state variables can be used to predict the position, velocity, etc., of the tracked object in the current frame. The predicted position, velocity, etc., of the tracked object can then be compared with the position, velocity, etc., of the detected object in the current frame to see the probability that each of the tracked and detected objects is the same object.

[0004] In such an approach, associations can sometimes fail and tracking can be lost. Therefore, identity switching can occur during tracking, causing the tracking to initially follow one object and then incorrectly continue following another. Summary of the Invention

[0005] The purpose of this invention is to promote enhanced correlation between object detection in the first frame and object detection in the second frame, so as to reduce the risk of erroneous correlation.

[0006] According to a first aspect, a method is provided for associating object detections in a first frame with object detections in a second frame using a convolutional neural network. The convolutional neural network is trained to determine feature vectors such that the feature vectors of object detections associated with individual objects are arranged in separate clusters in a feature vector space. In this method, regions corresponding to object detections in the first frame are received, and a set of feature vectors comprising two or more feature vectors associated with the regions in the first frame is determined by the trained convolutional neural network, thereby determining a feature vector reference set comprising two or more feature vectors. Further, multiple regions corresponding to corresponding object detections in a plurality of object detections in the second frame are received. For each of the multiple regions in the second frame, a set of feature vectors associated with that region in the second frame is determined by the trained convolutional neural network, thereby determining a plurality of feature vector candidate sets. For each feature vector in the feature vector reference set, among the feature vectors in the plurality of feature vector candidate sets, a predetermined number of feature vectors having the minimum distance in the feature vector space to that feature vector in the feature vector reference set are identified, thereby identifying the closest feature vector set. For each of the multiple feature vector candidate sets, the measure of proximity to the feature vector reference set is determined as the proportion of the closest feature vector sets belonging to that candidate set. For each of the multiple object detections in the second frame, weights are assigned based on the determined measure of proximity between the corresponding feature vector candidate set and the feature vector reference set. Based on the assigned weights, the object detection in the first frame is associated with one of the multiple object detections in the second frame.

[0007] As used in this article, the same object refers to the same instance of an object (i.e., the same physical object).

[0008] Object detection associated with a single object refers to object detection that is not associated with the same instance of the object (i.e., object detection associated with different physical objects). For example, it can be object detection associated with different instances of the same object type. In the case of multiple objects of different types, it can also be object detection associated with objects of different types.

[0009] In object detection, feature vectors associated with a single object are arranged in a separate cluster in the feature vector space. This means that the internal distance between feature vectors of the same object in the feature vector space is generally smaller than the distance between feature vectors of different objects. However, outliers can occur, meaning that the distance between an object's feature vector and other feature vectors of the object in the vector space may be much larger than the internal distance between other feature vectors of the object. In such cases, the distance of that feature vector to one or more feature vectors of other objects may be smaller than the distance to other feature vectors of its associated objects.

[0010] According to the method of the first aspect, an efficient way of setting weights is achieved by determining a proximity metric for each candidate set and then assigning a corresponding weight to each object detection based on the determined proximity metric of the candidate set corresponding to the corresponding object detection. This weight indicates the similarity between each object detection in the second frame and the object detection in the first frame. Such weights can, for example, be used to add attribute similarity relative to the appearance of an object associated with one of a plurality of object detections in the second frame, and thus enhance the algorithm, where such association is made on another basis (e.g., based on the predicted state of the object in the second frame). Furthermore, since the weights are based on a proximity metric, which in turn considers all feature vectors of the object detections in the first frame, the spatial variation of the feature vectors of the object detections in the first frame is taken into account in the assigned weights.

[0011] By selecting a higher predetermined number of eigenvectors with minimum distances in the eigenvector space, the influence on the proximity measure of a few outliers in the candidate set is reduced, so that the proximity measure will depend more on the eigenvectors of each candidate set that are closer to the center of that candidate set in the eigenvector space. On the other hand, the predetermined number of eigenvectors with minimum distances in the eigenvector space should not become so high that approaching it becomes close to the total number of eigenvectors in all candidate sets.

[0012] The assigned weights ensure that for any two object detections in multiple object detections within the second frame that have the same proximity metric, both object detections are assigned the same weight, and for any object detection in the second frame that has a higher proximity metric than another object detection in the second frame, the weight assigned to that object detection is higher than the weight assigned to the other object detection. Thus, when the proximity metrics differ, the weights affect the association, while when the proximity metrics are the same, the weights do not affect the association. This is advantageous, for example, when the weights are used to adjust the algorithm, associations are made on another basis (e.g., based on the predicted state of the object in the second frame). The weights can then be applied such that they add attribute similarity to the object's appearance in the association, making it more likely to associate with an object when the weights indicate that the object detection in the second frame has a higher proximity metric. On the other hand, when two different objects have the same weight, the weights do not affect the association, and the association will only be based on another basis (e.g., based on the predicted state).

[0013] The assigned weights can be such that, for each of the multiple object detections in the second frame, the assigned weight is proportional to a measure of proximity between the candidate set of feature vectors corresponding to that object detection and the reference set of feature vectors. This is advantageous, for example, when the weights are used to adjust the algorithm, associations are made on another basis (e.g., based on the predicted state of the objects in the second frame). The weights can then be applied such that they will add attribute similarity of object appearance in the association, so that the weights will affect the association proportionally to their size.

[0014] The assigned weights are such that for each of the multiple object detections in the second frame, the assigned weight is 1 plus the difference between the proximity metric of that object detection and the average proximity metric of the multiple object detections in the second frame excluding that object detection.

[0015] For each of the multiple object detections in the second frame, an association score can be calculated based on the weight of that object detection. This association score indicates the probability that the object detection in the second frame is associated with the object detection in the first frame.

[0016] The association score can be further based on the predicted state of the object detection in the second frame in the first frame.

[0017] The predicted state can include one or more of the following: predicted size, predicted shape, predicted position, predicted velocity, and predicted acceleration.

[0018] Object detections in the first frame can be associated with the object detection with the highest association score among multiple object detections in the second frame.

[0019] Object detections in the first frame can be associated with tracking, and tracking can be updated based on object detections in the second frame associated with those object detections in the first frame. By basing the association between object detections in the first frame and one of multiple object detections in the second frame on assigned weights, attribute similarity in object appearance is added, thus enhancing the algorithm where such associations are made on another basis (e.g., based on the predicted state of the object in the second frame). Therefore, updating tracking based on such associations reduces the risk of incorrectly updating tracking based on erroneous object detections in the second frame, and consequently the risk of tracking being lost or starting to follow another object.

[0020] Determining the set of feature vectors associated with regions in a first frame using a trained convolutional neural network may include: determining a first plurality of feature vectors for multiple sub-regions of the first image frame using the trained convolutional neural network; and identifying a set of feature vectors comprising two or more feature vectors from among the first plurality of feature vectors associated with regions in the first frame, thereby determining a reference set of feature vectors comprising two or more feature vectors. For each of the plurality of regions in a second frame, determining the set of feature vectors associated with that region in the second frame using a trained convolutional neural network may include: determining a second plurality of feature vectors for multiple sub-regions of the second image frame using the trained convolutional neural network; and for each of the plurality of regions in the second frame, identifying a set of feature vectors associated with that region in the second frame, thereby determining a plurality of candidate sets of feature vectors.

[0021] According to a second aspect, a non-transitory computer-readable storage medium is provided, having instructions stored thereon that, when executed by a processing device, implement the method according to the first aspect.

[0022] The aforementioned features of the method according to the first aspect also apply to the second aspect where applicable. To avoid undue repetition, please refer to the above.

[0023] According to a third aspect, there is provided an apparatus for associating object detections in a first frame with object detections in a second frame using a convolutional neural network trained to determine feature vectors, such that the feature vectors of object detections associated with individual objects are arranged in separate clusters in a feature vector space. The apparatus includes circuitry configured to perform the method according to the first aspect.

[0024] The aforementioned features of the method according to the first aspect also apply to this third aspect where applicable. To avoid undue repetition, please refer to the above.

[0025] Further applicability of the invention will become apparent from the detailed description given below. However, it should be understood that while the detailed description and specific embodiments indicate preferred embodiments of the invention, they are given by way of illustration only, as various variations and modifications within the scope of the invention will become apparent to those skilled in the art from this detailed description.

[0026] Therefore, it should be understood that the present invention is not limited to the specific components of the described apparatus or the operation of the described method, as such apparatus and methods can vary. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It must be noted that the articles “a,” “an,” “the,” and “the” used in the specification and appended claims are intended to indicate the presence of one or more elements unless the context clearly indicates otherwise. Thus, for example, references to “unit” or “the (described) unit” can include multiple devices, etc. Furthermore, the words “comprising,” “including,” “containing,” and similar terms do not exclude other elements or steps. Attached Figure Description

[0027] The above and other aspects of the invention will now be described in more detail with reference to the accompanying drawings. The drawings should not be considered limiting, but rather for explanation and understanding.

[0028] Figure 1 A flowchart is shown relating to an embodiment of the method of this disclosure that uses a convolutional neural network to associate object detection in a first frame with object detection in a second frame.

[0029] Figure 2a An illustration showing the first frame, including the objects.

[0030] Figure 2b An illustration of the second frame, which includes multiple objects, is shown.

[0031] Figure 3 Showing the relationship between two-dimensional space and Figure 2a The objects in the first frame shown in the figure and Figure 2b The figure shows a simplified representation of the feature vectors corresponding to multiple objects in the second frame.

[0032] Figure 4 The diagram illustrates an embodiment of the apparatus of this disclosure that uses a convolutional neural network to associate object detection in a first frame with object detection in a second frame. Detailed Implementation

[0033] The invention will now be described below with reference to the accompanying drawings, which illustrate presently preferred embodiments of the invention. However, the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0034] This invention applies to scenarios where object detection in a first frame is associated with object detection in a second frame. For example, such a scenario occurs when tracking objects in a frame sequence. In such tracking, the tracking associated with the object identifier in the first frame of the frame sequence should be updated relative to a second subsequent frame that includes multiple object detections. In this case, it is important that the object detection in the first frame is associated with a correct object detection from among the multiple object detections in the second frame.

[0035] Now relative to Figure 1 Together Figure 2a and Figure 2b as well as Figure 3 Together, we describe an embodiment of a method for associating object detection in a first frame with object detection in a second frame using a convolutional neural network. Figure 1 Flowchart 100 is shown. Figure 2a and Figure 2b The illustration shows a first frame 200 including object 210 and a second frame 220 including multiple objects (i.e., first object 230, second object 240, and third object 250), and Figure 3 Showing the relationship between two-dimensional space and Figure 2a The objects in the first frame 200 shown in the figure and Figure 2b The figure shows a simplified representation of the feature vectors corresponding to multiple objects in the second frame 220.

[0036] It is possible to perform operations on a frame sequence regarding Figure 1 The illustrated method involves several steps. Therefore, this method can be applied to a previous frame (as the first frame) and a second frame (as the second frame). Figure 2a The first frame, 200, is used for execution. Similarly, the method can be subsequently executed relative to the first frame. Figure 2b The second frame 220 and another subsequent frame as the second frame are executed.

[0037] Convolutional neural networks are trained to determine feature vectors such that the feature vectors of object detections associated with individual objects are arranged in separate clusters in the feature vector space. Object detection associated with individual objects refers to object detection that is not related to the same instance of the object. For example, an individual object can be a different instance of the same object type or an object of a different type.

[0038] Feature vectors for a frame can be determined first by identifying a feature vector for each of multiple sub-regions of the frame, where these sub-regions preferably together form the complete frame. For example, the frame can be divided into a grid of n×n pixel sub-regions, where n is a non-zero positive integer. For instance, a feature vector can be determined for each pixel in the frame, but typically the spatial resolution of the feature vectors is lower than this spatial resolution (e.g., one feature vector per 8×8 pixels). After the feature vectors for the entire frame have been determined, the feature vectors for object detection of the frame can then be determined by identifying the feature vectors of the sub-regions of the frame within the region corresponding to object detection.

[0039] Training can be accomplished using any known training method that achieves the desired characteristics of a convolutional neural network, where feature vectors for object detection associated with individual objects are arranged in separate clusters in the feature vector space. Such training can, for example, be based on a loss function that encourages the convolutional neural network to map each sub-region of a frame to feature vectors in the feature vector space, such that sub-regions belonging to the same object (i.e., the same instance of the same object type) are mapped to feature vectors that are close together in the feature vector space, while sub-regions belonging to different objects (i.e., different instances of the same object type or different object types) are mapped to feature vectors that are far apart in the feature vector space. The distance between feature vectors can, for example, be limited to Euclidean distance in the feature vector space. Such a loss function can comprise the sum of all pairs of sub-regions for each of multiple training frames. Each term in the sum can, for example, include a first term belonging to a pair of sub-regions of the same object and a second term belonging to a pair of sub-regions of different objects. The first term contributes a small amount to the low values ​​of the distance between the corresponding feature vectors and a large amount to the high values ​​of the distance between the corresponding mapped feature vectors, and the second term contributes a small amount to the high values ​​of the distance between the corresponding feature vectors and a large amount to the low values ​​of the distance between the corresponding feature vectors. Each term in the sum can be further normalized relative to the total number of pixels in the corresponding training image, and the weights are weighted by the corresponding pixel and the pixels of the training image. Kong, Shu, et al. from the Department of Computer Science at the University of California provided an example of a usable neural network and its training method in "Recurrent PixelEmbedding for Instance Grouping" on December 22, 2017 (arXiv:1712.08273v1). De Brabandere et al. (ESAT-PSI, KU Leuven, Belgium) provided another example of a usable neural network in “Semantic Instance Segmentation with a Discriminative Loss Function” on August 8, 2017 (arXiv: 1708.02551v1). Fathi et al., Google Inc. and UCLA provided yet another example in “Semantic Instance Segmentation via Deep Metric Learning” on March 30, 2017 (arXiv: 1703.10277v1). For the purposes of this application, training can be performed using the COCO dataset (https: / / cocodataset.org / #home).

[0040] For embodiments involving objects of different types, the convolutional neural network needs to be trained using images that include objects of different types, such that pixels belonging to different object types are mapped to feature vectors that are far apart in the feature vector space. This can be achieved, for example, by training any of the aforementioned neural networks using the COCO dataset.

[0041] In this method, the region corresponding to the object detection in the first frame is received in S110. For example, in Figure 2a In the first frame 200 illustrated in the figure, object 210 has been detected. The region corresponding to this object detection is the region of object 210 with a specific size, shape, and position in the first frame 200. Object detection can be determined, for example, by an algorithm for object instance segmentation that provides information about different object detections and positions in the image. The information about the position can be in the form of a mask, which can be used to mask the region corresponding to the object detection. An example of an object instance segmentation algorithm that can be used is Masked R-CNN (arXiv:17.03.06870v3), described in the paper "Mask R-CNN" published by He et al. on January 14, 2018. This paper describes the framework of Masked R-CNN, how it is trained, and suitable training data.

[0042] The feature vector reference set is determined S120 as a set of feature vectors associated with the receiving region in the first frame. For this purpose, a convolutional neural network trained as described above can be used. The complete set of feature vectors for the first frame can be determined by determining a feature vector for each of multiple sub-regions of the frame, wherein the multiple sub-regions preferably together form a complete frame. The feature vector reference set can then be determined by identifying the feature vectors of the complete set of feature vectors associated with the feature vectors of the sub-regions located within the receiving region of the first frame. Alternatively, for the first frame, it is sufficient to determine only the feature vectors associated with the receiving region. The feature vector reference set can then be determined by determining the feature vectors of the sub-regions located within the receiving region of the first frame. Relative to... Figure 2a In the first frame 200 illustrated in the figure, the feature vector reference set of object 210 is determined to be the set of feature vectors of sub-regions located within the region of object 210.

[0043] The feature vector reference set comprises two or more feature vectors. This can be achieved by selecting the size of the sub-regions that determine the corresponding feature vectors, such that the region corresponding to object detection comprises at least two sub-regions. Alternatively or additionally, association is only permitted for objects that have at least a minimum size relative to the size of the sub-regions, such that each object has at least a specific number of feature vectors. This is advantageous because objects need to have a reasonable size in order to successfully associate (re-identify). Furthermore, the more feature vectors an object has, the more spatial variations of the object can be considered, and the more statistical basis is available for association. The specific number of feature vectors can be two, but is typically larger than that.

[0044] Further receive multiple regions in the second frame of S130 that correspond to the corresponding object detections in the multiple object detections in the second frame. For example, in Figure 2b In the second frame 220 illustrated, objects 230, 240, and 250 have been detected. The regions corresponding to these object detections are areas in the second frame 220 containing objects 230, 240, and 250 with specific sizes, shapes, and positions. These object detections can also be determined using object segmentation algorithms such as the masked R-CNN described above.

[0045] Then, multiple feature vector candidate sets are determined S140 as multiple feature vector sets associated with multiple regions received in the second frame. For this, a trained convolutional neural network can be used. As described relative to the first frame, a complete set of feature vectors can also be determined for the second frame. Then, for each object detection in the multiple object detections of the second frame, a feature vector candidate set can be determined by identifying feature vectors of the complete set of feature vectors associated with the sub-regions located within the region corresponding to that object detection. Alternatively, for the second frame, it is sufficient to determine only the feature vectors associated with the multiple received regions. Then, for each of the multiple received regions, a feature vector candidate set can be determined by determining the feature vectors of the sub-regions located within that received region. Relative to... Figure 2b In the second frame 220 illustrated in the figure, for each of the multiple objects 230, 240, 250, a multiple set of feature vector candidates is determined as a set of feature vectors of sub-regions located within the region of that object.

[0046] Then, for each feature vector in the reference feature vector set, the closest feature vector set of S150 is identified by recognizing a predetermined number of feature vectors from multiple candidate feature vector sets, where the predetermined number of feature vectors have the minimum distance to that feature vector in the reference set in the feature vector space. For this purpose, the same distance metric used when training convolutional neural networks can be used, although other options may exist. Therefore, for each feature vector in the reference set, the distance to each feature vector in the multiple candidate feature vector sets is determined, and the predetermined number of feature vectors with the minimum distance from the multiple candidate feature vector sets are included in the closest feature vector set.

[0047] The predetermined number of feature vectors is preferably less than or equal to the number of feature vectors in the candidate set with the minimum number of feature vectors. For example, if a specific lower bound is set on the number of feature vectors for each object, the predetermined number can be set below that specific lower bound. This is advantageous because all feature vectors of a single object can be found in the feature vector space that are closest to each vector in the reference set.

[0048] The predetermined number of eigenvectors can be set to 1, but setting it to a larger number may be advantageous because it reduces the influence of outliers, i.e., the influence of eigenvectors in the reference set and candidate set being far apart from other eigenvectors in the same reference set or candidate set in the eigenvector space. For example, the predetermined number of eigenvectors can be set as a specific percentage of the number of eigenvectors in the candidate set with the minimum number of eigenvectors.

[0049] Figure 3 Showing the relationship between two-dimensional space and Figure 2a Object 210 and in the first frame 200 shown in the figure Figure 2b The figure shows a simplified representation of the feature vectors corresponding to the multiple objects 230, 240, and 250 in the second frame 220. The feature vectors are represented as different geometric shapes of points in two-dimensional space. For Figure 2a The object 210 in the first frame 200 illustrated in the figure has already had a feature vector reference set consisting of four feature vectors determined. Figure 3 In this context, the four eigenvectors—composed of the first eigenvector 312, the second eigenvector 314, the third eigenvector 316, and the fourth eigenvector 318 of the reference set—are represented as four circles. For Figure 2b The first object 230 in the second frame 220 illustrated in the figure has been identified as having a first candidate set of feature vectors consisting of three feature vectors. Figure 3 In the first candidate set, the three feature vectors consisting of the first feature vector 332, the second feature vector 334, the third feature vector 336, and the fourth feature vector 338 are represented by four plus signs. Figure 2bThe second object 240 in the second frame 220 illustrated in the figure has been determined to have a second candidate set of feature vectors consisting of four feature vectors. Figure 3 In this context, the four eigenvectors formed by the first eigenvector 342, the second eigenvector 344, the third eigenvector 346, and the fourth eigenvector 348 of the second candidate set are represented as four squares. For Figure 2b The third object 250 in the second frame 220 illustrated in the figure has been identified as a second candidate set of feature vectors consisting of four feature vectors. Figure 3 In the diagram, the four eigenvectors formed by the first eigenvector 352, the second eigenvector 354, the third eigenvector 356, and the fourth eigenvector 358 of the second candidate set are represented as four triangles. The number of eigenvectors in both the reference set and each candidate set is four. This is for illustrative purposes only. Typically, the number of eigenvectors can be ten, hundreds, or more. Furthermore, the number of eigenvectors in the reference set can differ from the number of eigenvectors in each candidate set, and the number of eigenvectors between candidate sets can also differ. For a predetermined number of eigenvectors with a minimum distance equal to 4, and where the distance is the Euclidean distance in two-dimensional space, the eigenvectors with the minimum distance to each eigenvector in the reference set are as follows: For the first eigenvector 312 of the reference set, the four closest eigenvectors in the candidate sets are the first eigenvector 332 of the first candidate set and the first eigenvectors 342, the second eigenvector 344, and the third eigenvector 346 of the second candidate set. For the second eigenvector 314 of the reference set, the four closest eigenvectors in the candidate sets are the first eigenvector 342 and the second eigenvector 344 of the second candidate set and the first eigenvector 352 and the second eigenvector 354 of the third candidate set. For the third eigenvector 316 of the reference set, the four closest eigenvectors of the candidate set are the first eigenvector 332 of the first candidate set, the first eigenvector 342 and the second eigenvector 344 of the second candidate set, and the first eigenvector 352 of the third candidate set. For the fourth eigenvector 318 of the reference set, the four closest eigenvectors of the candidate set are the first eigenvector 342, the second eigenvector 344, the third eigenvector 346, and the fourth eigenvector 348 of the second candidate set. This is summarized in Table 1 below.

[0050]

[0051] Table 1

[0052] Back Figure 1 For each candidate set of feature vectors in multiple candidate feature vector sets, a measure of proximity between S160 and the reference set of feature vectors is determined. The measure of proximity is determined as the proportion of the closest feature vector set belonging to that candidate feature vector set.

[0053] As shown in Table 1, the total number of eigenvectors in the closest eigenvector set of the candidate set is 16. For the first candidate set, the first eigenvector 332 is included twice in the closest eigenvector set. Therefore, the proximity measure of the first candidate set is 2 / 16. For the second candidate set, the first eigenvector 342 and the second eigenvector 344 are both included four times, the third eigenvector 346 is included twice, and the fourth eigenvector is included once in the closest eigenvector set. Therefore, the proximity measure of the second candidate set is 11 / 16. For the third candidate set, the first eigenvector 352 is included twice, and the second eigenvector 354 is included once in the closest eigenvector set. Therefore, the proximity measure of the third candidate set is 3 / 16.

[0054] S170 weights are assigned to each of the multiple object detections in the second frame. Each S170 weight is assigned based on a metric of proximity between the candidate feature vector set corresponding to that object detection and the reference feature vector set. Therefore, relative to... Figure 2b The first weight assigned to the first object 230 is based on a proximity metric determined by the first candidate set, the second weight assigned to the second object 240 is based on a proximity metric determined by the second candidate set, and the third weight assigned to the third object 250 is based on a proximity metric determined by the third candidate set.

[0055] Because the assigned weights are based on a measure of proximity to a reference set corresponding to the object detections in the first frame and the object detections in the second frame, and on how that proximity measure is determined, the assigned weights typically indicate a measure of appearance similarity between objects in the first frame and each object in the second frame. These weights can also be referred to as appearance-based weights.

[0056] By selecting a higher predetermined number of feature vectors that have the minimum distance in the feature vector space, the influence on the proximity measure of a few outliers in the candidate set is reduced, so that the proximity measure will depend more on the feature vectors located closer to the center of the candidate set in the feature vector space.

[0057] The weights are preferably assigned such that if two object detections in a plurality of object detections in a second frame have the same proximity metric, then the weights assigned to each of them are the same, and if the first object detection has a higher proximity metric than the second object detection, then the weight assigned to the first object detection is higher than the weight assigned to the second object detection. Furthermore, the weights assigned to object detections can be proportional to the determined proximity metric of the corresponding candidate set.

[0058] Then, based on the assigned weights, the object detection in the first frame is associated with one of the multiple object detections in the second frame (S180). For example, the assigned weights can be used to calculate the association score for each combination of the object detection in the second frame and the object detection in the first frame (S175). Each association score indicates the probability that the relevant object detection in the second frame and the object detection in the first frame involve the same object, and is based on the weight of the relevant object detection in the second frame relative to the object detection in the first frame. Then, the object detection in the first frame can be associated with the object detection with the highest association score among the multiple object detections in the second frame (S180).

[0059] For example, if the weights and association scores are based solely on the appearance similarity between each of the multiple object detections in the second frame and the object in the first frame, then the weight of each object detection in the second frame can be set equal to a measure of its proximity to the object detections in the first frame. Then, the association score of each object detection in the second frame relative to the object detections in the first frame can be set equal to its weight.

[0060] for Figure 2b In the second frame 220 illustrated, objects 230, 240, and 250, based on the proximity metrics determined in Table 1, have the following weights and association scores: First object 230 has a weight and association score of 2 / 16; Second object 240 has a weight and association score of 11 / 16; Third object 250 has a weight and association score of 3 / 16.

[0061] Each of the association scores can be further based on the predicted state of the object detection in the first frame in the second frame (e.g., based on Kalman filter state variables). The predicted state can include one or more of predicted size, predicted shape, predicted position, predicted velocity, and acceleration. The predicted state of the object detection in the first frame in the second frame is typically based on the tracking involved in the object detection in the first frame. After association with the object detection, the tracking is updated in each frame with the determined state. It is important to note that the determined state after association does not necessarily have to be identical to the features of the associated object detection. For example, the determined state may differ from the state of the associated object detection in one or more aspects of size, shape, position, velocity, and acceleration. The predicted state of the object detection in the first frame in the second frame can be compared with each of the object detections in the second frame to determine the probability that the object detection in the first frame and the related object detection in the second frame involve the same object. For example, distance d j This can be determined as the difference between the predicted state of the object detection in the first frame and the j-th object detection in the second frame. Distance d jIt can be simply based on the predicted state of object detection in the first frame in the second frame and the distance between the center point of the object detected in the j-th object in the second frame, or it can be based on the overlap of objects, based on the predicted state of object detection in the first frame in the second frame and the j-th object detection in the second frame, for example, as measured by the joint intersection.

[0062] Distance d j This can be converted into the probability that an object detection in the first frame corresponds to the j-th object detection in the second frame. For example, a Gaussian curve or other curves that decay with increasing distance can be used to convert distance into probability. By way of example, the probability P of an object detection in the first frame corresponding to the j-th object detection in the second frame is... j The following equation can be used to calculate:

[0063]

[0064] Where σ is a predetermined constant.

[0065] Distance d j It can be based on appearance-based weights w j To modify, based on appearance weight w j It is calculated based on a proximity metric. For example, distance can be divided by a weight and may be rescaled by a constant. Based on such a modification of distance, the probability of modification or association score can be calculated according to the following equation:

[0066]

[0067] Where α is a constant used to manage the influence of appearance-based weights.

[0068] When using appearance-based weights to modify probabilities based on predicted states, for each of the multiple object detections in the second frame, the assigned weight can be such that the assigned weight is 1 plus the difference between the proximity metric of that object detection and the average proximity metric of the other object detections in the second frame. Therefore, if there are m object detections in the second frame and the proximity metric of the j-th object detection in the second frame to the object detections in the first frame is denoted as a... j Then the weight w for detecting the j-th object j Calculate based on the following equation:

[0069]

[0070] for Figure 2bBased on the objects 230, 240, and 250 in the second frame 220 illustrated in the figure, and the proximity metric determined based on Table 1, the weight w1 of the first object 230 according to Equation 2 is 1 + (2 / 16 - 1 / 2 * (11 / 16 + 3 / 16)) = 22 / 32. According to Equation 1, the weight w2 of the second object 240 is 1 + (11 / 16 – 1 / 2 * (2 / 16 + 3 / 16)) = 49 / 32. According to Equation 1, the weight w3 of the third object 250 is 1 + (3 / 16 - 1 / 2 * (2 / 16 + 11 / 16)) = 25 / 32.

[0071] Then, the object detection in the first frame can be associated with the object detection with the highest modification probability (i.e., the highest association score) among multiple object detections in the second frame S180.

[0072] The association can, for example, be used to update tracking involving object detections in the first frame based on multiple object detections in the second frame. Then, the S185 tracking can be updated based on object detections in the second frame associated with object detections in the first frame.

[0073] The above has described how an object detection in the first frame is associated with an object detection in a plurality of object detections in the second frame. However, the method of this disclosure is also applicable to the case where multiple object detections in the first frame are associated with multiple object detections in the second frame. In such a scenario, corresponding weights are assigned to each of the multiple object detections in the second frame relative to each object detection in the first frame.

[0074] Then, multiple object detections in the first frame can be associated with corresponding object detections in multiple object detections in the second frame based on assigned weights (S180). For example, the assigned weights can be used to calculate the association score for each combination of an object detection in the multiple object detections in the second frame and an object detection in the multiple object detections in the first frame (S175). Each association score indicates the probability that the relevant object detections in the second frame and the relevant object detections in the first frame involve the same object, and is based on the weight of the relevant object detections in the second frame relative to the relevant object detections in the first frame. The association scores can then be used in an algorithm that associates each object detection in the multiple object detections in the first frame with at most one object detection in the multiple object detections in the second frame, such that each object detection in the second frame is associated with at most one object detection in the multiple object detections in the first frame. For example, if the Hungarian algorithm is used to associate each object detection in a first frame (previous frame) with at most one object detection in a second frame (current frame), the input to the Hungarian algorithm is a probability, representing the likelihood that for each object detection in the second frame, the object detection in the first frame is associated with that object detection in the second frame (i.e., they are the same object). Therefore, the association score can be calculated as such a probability. For example, if the weights and association scores are based solely on the appearance similarity between each of the multiple objects in the second frame and each of the multiple objects in the first frame, then the weight of each object detection in the second frame relative to each object detection in the first frame can be set equal to the corresponding measure of proximity of each object detection in the second frame relative to each object detection in the first frame. Then, the association score of each object detection in the second frame relative to each object detection in the first frame can be set equal to the corresponding weight of each object detection in the second frame relative to each object detection in the first frame.

[0075] Additionally, the association algorithm may include a threshold below which object detections in the first frame are not associated with any object detections in the second frame. In such cases, the tracking associated with object detections in the first frame is referred to as lost tracking. Tracking can then be terminated, or a timeout can be used, where tracking is updated only based on the predicted state, without using any associated object detections in the second frame to see if association can be made in subsequent frames. Furthermore, the algorithm can further support the creation of corresponding new tracking for object detections in the second frame that are not associated with any object detections in the first frame. To avoid creating new tracking based on object detections in the second frame that only appear in one or a few frames and are due to noise, a delay can be used such that new tracking is only created if object detection continues for more than a predetermined number of frames.

[0076] Each of the association scores can be further based on the predicted state of the corresponding object detection in the second frame (e.g., based on Kalman filter state variables) of the multiple object detections in the first frame. The predicted state can include one or more of predicted size, predicted shape, predicted position, predicted velocity, and acceleration. The predicted state of the corresponding object detection in the second frame is typically based on the tracking involved in the corresponding object detection in the first frame. After association with the object detection, the tracking is updated in each frame with the determined state. It is important to note that the determined state after association does not necessarily have to be identical to the features of the associated object detection. For example, the determined state may differ from the state of the associated object detection in one or more aspects of size, shape, position, velocity, and acceleration. For each object detection in the first frame, the predicted state of that object detection in the second frame can be compared with the object detection in the second frame to determine the probability that the object detection in the first frame and the associated object detection in the second frame involve the same object. For example, distance d kj This can be determined as the difference between the predicted state of the k-th object detection in the first frame and the predicted state of the j-th object detection in the second frame. Distance d kj It can be simply based on the distance between the predicted state of the k-th object detection in the first frame and the center point of the object detected in the j-th object detection in the second frame, or it can be based on the overlap of objects, based on the predicted state of the k-th object detection in the first frame and the j-th object detection in the second frame, for example, as measured by the joint intersection.

[0077] Distance d kjThis can be converted into the probability that the k-th object detection in the first frame corresponds to the j-th object detection in the second frame. For example, a Gaussian curve or other curves that decay with increasing distance can be used to convert distance into probability. By way of example, the probability P of the k-th object detection in the first frame corresponding to the j-th object detection in the second frame is... kj The following equation can be used to calculate:

[0078]

[0079] Where σ is a predetermined constant.

[0080] Distance d kj Appearance-based weights w can be calculated based on proximity-based metrics. kj This can be modified. For example, the distance can be divided by the weight and possibly rescaled by a constant. Based on such modifications to the distance, the probability of modification or the association score can be calculated according to the following equation:

[0081]

[0082] Where α is a constant used to manage the influence of appearance-based weights.

[0083] When using appearance-based weights to modify probabilities based on predicted states, the weights assigned to multiple object detections in the first frame can all be calculated in the same manner as described relative to the weights assigned to a single object detection in the first frame. For example, if there are m object detections in the second frame and the measure of proximity between the j-th object detection in the second frame and the k-th object detection in the first frame is denoted as a... kj Then the weight w of the k-th object detection in the first frame relative to the j-th object detection in the second frame kj Calculate based on the following equation:

[0084]

[0085] Then, each object detection in the first frame can be associated with the object detection in the second frame that has the highest association score (S180). Alternatively, the association scores can then be used in an algorithm that associates each object detection in the first frame with at most one object detection in the second frame, such that each object detection in the second frame is associated with at most one object detection in the first frame. For example, if the Hungarian algorithm is used, the association scores can be determined based on assigned weights and prediction states, such that they indicate probabilities P. kj For example, according to Equation 5, the probability P kjIndicates the probability that the k-th object detection in the first frame is associated with the j-th object detection among multiple object detections in the second frame (i.e., they are the same object).

[0086] The association can, for example, be used to update tracking involving multiple object detections in the first frame based on multiple object detections in the second frame. For each object detection in the first frame, the tracking corresponding to S185 can then be updated based on the object detections in the second frame associated with the object detections in the first frame.

[0087] Figure 4 The diagram illustrates an embodiment of the apparatus 400 of this disclosure that uses a convolutional neural network to correlate object detection in a first frame with object detection in a second frame. Apparatus 400 includes circuitry 410. Circuitry 410 is configured to perform the functions of apparatus 400. Circuitry 410 may include a processor 412, such as a central processing unit (CPU), microcontroller, or microprocessor. Processor 412 is configured to execute program code. The program code may, for example, be configured to perform the functions of apparatus 400.

[0088] Device 400 may further include memory 430. Memory 430 may be one or more of a buffer, flash memory, hard disk drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or other suitable devices. In a typical arrangement, memory 430 may include non-volatile memory for long-term data storage and volatile memory serving as system memory for circuit 410. Memory 430 may exchange data with circuit 410 via a data bus. Accompanying control lines and address buses may also exist between memory 430 and circuit 410.

[0089] The functionality of the image processing apparatus 400 can be implemented in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory 430) of the apparatus 400 and executed by the circuitry 400 (e.g., using processor 412). Furthermore, the functionality of the apparatus 400 can be a standalone software application or part of a software application that performs additional tasks associated with the apparatus 400. The described functionality can be considered as a method configured to be executed by a processing unit (e.g., processor 412 of circuitry 410). Additionally, while the described functionality can be implemented in software, such functionality can also be implemented by dedicated hardware or firmware, or some combination of hardware, firmware, and / or software.

[0090] Circuit 410 is configured to perform a first region receiving function 431 and a reference set determination function 432. The first region receiving function 431 is configured to receive a region in the first frame corresponding to the object detection in the first frame. The reference set determination function 432 is configured to determine a feature vector reference set including two or more feature vectors associated with the region in the first frame by using a trained convolutional neural network.

[0091] Circuit 410 is further configured to perform a second region receiving function 433 and a candidate set determination function 434. The second region receiving function 433 is configured to receive multiple regions of the second frame corresponding to the corresponding object detections of multiple object detections in the second frame. The candidate set determination function 434 is configured to determine, for each of the multiple regions in the second frame, a set of feature vectors associated with that region in the second frame by a trained convolutional neural network, thereby determining multiple feature vector candidate sets.

[0092] Circuit 410 is further configured to perform the closest feature vector identification function 435, which is configured to, for each feature vector in the feature vector reference set, identify a predetermined number of feature vectors in the feature vector space that have the minimum distance to the feature vector in the feature vector reference set among the feature vectors in a plurality of feature vector candidate sets, thereby identifying the closest feature vector set.

[0093] Circuit 410 is further configured to perform a proximity metric determination function 436, which is configured to determine, for each of a plurality of feature vector candidate sets, the proximity metric with the feature vector reference set as the proportion of the closest feature vector set belonging to that feature vector candidate set.

[0094] Circuit 410 is further configured to perform a weight allocation function 437, which is configured to allocate weights for each of the plurality of object detections in the second frame based on a measure of proximity between the candidate set of feature vectors corresponding to the object detection and the reference set of feature vectors.

[0095] Circuit 410 is further configured to perform association function 439, which is configured to associate an object detection in the first frame with one of a plurality of object detections in the second frame based on assigned weights.

[0096] The weight allocation function 437 can be configured to allocate weights such that for any two object detections with the same proximity metric among the multiple object detections in the second frame, the weights allocated to the two object detections are the same, and for any object detection among the multiple object detections in the second frame that has a higher proximity metric than another object detection among the multiple object detections in the second frame, the weight allocated to the other object detection is higher.

[0097] Circuit 410 may be further configured to perform an association score calculation function 438, which is configured to calculate, for each of the plurality of object detections in the second frame, an association score based on the weight of that object detection, indicating the probability that the object detection in the second frame is associated with an object detection in the first frame. The association score may be further based on the prediction state of the object detection in the first frame in the second frame. The prediction state may include one or more of predicted size, predicted shape, predicted position, predicted velocity, and acceleration.

[0098] The association function 439 can be configured to associate object detections in the first frame with the object detections with the highest association scores among multiple object detections in the second frame.

[0099] Object detection in the first frame can be associated with tracking, and the circuit can be further configured to perform a tracking update function 440, which is configured to update the tracking based on object detection in the second frame that is associated with object detection in the first frame.

[0100] The function performed by circuit 410 can be further applied to... Figure 1 , Figure 2a and Figure 2b as well as Figure 3 The corresponding steps of an embodiment of the described method.

[0101] Those skilled in the art will understand that the present invention is not limited to the embodiments described above. Rather, many modifications and variations are possible within the scope of the appended claims. Such modifications and variations can be understood and implemented by those skilled in the art in practicing the claimed invention through a study of the drawings, the disclosure, and the appended claims.

Claims

1. A method of associating an object detection in a first frame with an object detection in a second frame such that feature vectors related to individual object detections are arranged in separate clusters in a feature vector space using a convolutional neural network trained to determine feature vectors, the method comprising: receiving a region of the first frame corresponding to the object detection in the first frame; determining, by the trained convolutional neural network, a set of feature vectors associated with the region in the first frame, thereby determining a reference set of feature vectors comprising two or more feature vectors; receiving a plurality of regions of the second frame corresponding to respective object detections in a plurality of object detections in the second frame; for each of the plurality of regions in the second frame, determining, by the trained convolutional neural network, a set of feature vectors associated with the region in the second frame, thereby determining a plurality of candidate sets of feature vectors, each comprising two or more feature vectors; for each feature vector of the reference set of feature vectors, identifying, among the feature vectors of the plurality of candidate sets of feature vectors, a predetermined number of feature vectors having a minimum distance to the feature vector of the reference set of feature vectors in the feature vector space, thereby identifying a closest set of feature vectors; for each candidate set of feature vectors of the plurality of candidate sets of feature vectors, determining a measure of closeness to the reference set of feature vectors as a proportion of the closest set of feature vectors belonging to the candidate set of feature vectors; for each of the plurality of object detections in the second frame, assigning a weight based on the determined measure of closeness to the reference set of feature vectors of the candidate set of feature vectors corresponding to the object detection; and associating the object detection in the first frame with one of the plurality of object detections in the second frame based on the assigned weights.

2. The method of claim 1, wherein: for any two object detections of the plurality of object detections in the second frame having the same measure of closeness, the assigned weights of the two object detections are the same; and for any object detection of the plurality of object detections in the second frame having a higher measure of closeness than another object detection of the plurality of object detections in the second frame, the assigned weight of the any object detection is higher than the assigned weight of the another object detection.

3. The method of claim 1, wherein: for each of the plurality of object detections in the second frame, the assigned weight is proportional to the determined measure of closeness to the reference set of feature vectors of the candidate set of feature vectors corresponding to the object detection.

4. The method of claim 1, wherein: ​ ​ For each of the plurality of object detections in the second frame, the assigned weight is one plus a difference between a measure of proximity of that object detection and an average of the measures of proximity of each of the object detections in the plurality of object detections in the second frame other than that object detection.

5. The method of claim 1, further comprising: For each of the plurality of object detections in the second frame, computing a association score indicative of a probability that the object detection in the second frame is associated with the object detection in the first frame based on the weight of that object detection.

6. The method of claim 5, wherein, The association score is further based on a predicted state of the object detection in the first frame in the second frame.

7. The method of claim 6, wherein, The predicted state comprises one or more of a predicted size, a predicted shape, a predicted location, a predicted velocity, and an acceleration.

8. The method of claim 1, wherein, The object detection in the first frame is associated with the object detection of the plurality of object detections in the second frame having the highest association score.

9. The method of claim 1, wherein, The object detection in the first frame is associated with a track, further comprising: updating the track based on the object detection in the second frame associated with the object detection in the first frame.

10. The method of claim 1, wherein, The act of determining, by the trained convolutional neural network, a set of feature vectors associated with the region in the first frame comprises: determining, by the trained convolutional neural network, a first plurality of feature vectors for a plurality of sub-regions of the first frame; and identifying, from the first plurality of feature vectors associated with the region in the first frame, a set of feature vectors comprising two or more feature vectors, thereby determining a reference set of feature vectors comprising two or more feature vectors; and wherein the act of determining, by the trained convolutional neural network, a set of feature vectors associated with each of the plurality of regions in the second frame comprises: determining, by the trained convolutional neural network, a second plurality of feature vectors for a plurality of sub-regions of the second frame; and For each of the plurality of regions in the second frame, identifying a set of feature vectors associated with that region in the second frame, thereby determining a plurality of candidate sets of feature vectors.

11. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a device having processing capability, implement the method of any one of claims 1 to 10.

12. An apparatus for associating an object detection in a first frame with an object detection in a second frame using a convolutional neural network trained to determine feature vectors such that feature vectors of object detections related to a single object are arranged in separate clusters in a feature vector space, the apparatus comprising circuitry configured to perform the following acts: receiving a region of the first frame corresponding to the object detection in the first frame; determining, by the trained convolutional neural network, a set of feature vectors associated with the region in the first frame comprising two or more feature vectors, thereby determining a reference set of feature vectors comprising two or more feature vectors; receive, for the second frame, a plurality of regions corresponding to respective ones of a plurality of object detections in the second frame; for each of the plurality of regions in the second frame, determine, by the trained convolutional neural network, a set of feature vectors associated with the region in the second frame, the set of feature vectors including two or more feature vectors, thereby determining a plurality of sets of feature vector candidates, each set of feature vector candidates including two or more feature vectors; for each feature vector of the set of feature vector references, identify, among the feature vectors of the plurality of sets of feature vector candidates, a predetermined number of feature vectors having a smallest distance to the feature vector of the set of feature vector references in the feature vector space, thereby identifying a set of closest feature vectors; for each set of feature vector candidates of the plurality of sets of feature vector candidates, determine a measure of closeness to the set of feature vector references as a proportion of the set of closest feature vectors that belong to the set of feature vector candidates; for each of the plurality of object detections in the second frame, assign a weight based on the determined measure of closeness to the set of feature vector references of the set of feature vector candidates corresponding to the object detection; and associate the object detection in the first frame with one of the plurality of object detections in the second frame based on the assigned weights.

13. The apparatus of claim 12, wherein: for any two of the plurality of object detections in the second frame having a same measure of closeness, the assigned weights of the two object detections are the same; and for any of the plurality of object detections in the second frame having a higher measure of closeness than another of the plurality of object detections in the second frame, the assigned weight of the any object detection is higher than the assigned weight of the other object detection.

14. The apparatus of claim 12, wherein, the circuitry is further configured to, for each of the plurality of object detections in the second frame, compute, based on the weight of the object detection, a correlation score indicative of a probability that the object detection in the second frame is associated with the object detection in the first frame.

15. The apparatus of claim 14, wherein, the correlation score is further based on a predicted state of the object detection in the first frame in the second frame.

Images