DEVICE AND METHOD FOR CORRELATING OBJECT DETECTION ACROSS FRAME USING NEURAL NETWORKS

Abstract

To provide a method and device which facilitates improvement in association of an object detection in a first frame with an object detection in a second frame in order to reduce risks of erroneous association.SOLUTION: The method for associating object detection between frames uses a convolutional neural network to: determine a reference set of two or more feature vectors associated with an identified area corresponding to an object detection in a first frame and a candidate set of feature vectors associated with a corresponding area of an identified area corresponding to an object detection in a second frame; determine measurement of closeness to the reference set of the feature vectors; determine corresponding weights on the basis of the determined measurement of closeness to the reference set of the feature vectors of the candidate set; and associate with one of the objects in the second frame on the basis of the assigned weights.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to tracking objects in video, and in particular to correlating an object detection in a first frame with an object detection in a second frame using a convolutional neural network. [Background technology]

[0002] When tracking multiple objects in video, a Kalman filter or similar method may be used. In such methods, the tracks of multiple objects are updated sequentially from frame to frame based on object detections in the frames. To update the track, the object detection in the current frame is first associated with an existing track in a previous frame. When the association is made, the track in the previous frame is updated based on the position of the associated object detection in the current frame.

[0003] The association may be made based on the probability of how likely it is that the existing track in the previous frame and the object detection in the current frame are the same object. These probabilities may be based on the Kalman filter state variables. For example, the Kalman filter state variables may be used to predict the position, velocity, etc. of the tracked object in the current frame. The predicted position, velocity, etc. of the tracked object can then be compared to the position, velocity, etc. of the detected object in the current frame to see how likely it is that the tracked object and each of the detected objects are the same object.

[0004] With such an approach, sometimes the association fails and the track is lost. As a result, the track may have an identity switch, whereby the track first follows one object and then erroneously continues to follow another object. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] "Recurrent Pixel Embedding for Instance Grouping" by Kong, Shu et al., Department of Computer Science, University of California, USA, December 22, 2017 (arXiv:1712.08273v1) [Non-Patent Document 2] "Semantic Instance Segmentation with a Discriminative Loss Function" by De Brabandere et al., ESAT-PSI, KU Leuven, Belgium, August 8, 2017 (arXiv:1708.02551v1) [Non-Patent Document 3] Fathi et al., Google Inc. and UCLA, "Semantic Instance Segmentation via Deep Metric Learning," USA, March 30, 2017 (arXiv:1703.10277v1) [Non-Patent Document 4] "Mask R-CNN" by He et al. from January 14, 2018 (arXiv:17.03.06870v3) Summary of the Invention

[0006] It is an object of the present invention to facilitate improved association of an object detection in a first frame with an object detection in a second frame in order to reduce the risk of erroneous association.

[0007] According to a first aspect, a method is provided for correlating an object detection in a first frame with an object detection in a second frame using a convolutional neural network. The convolutional neural network is trained to identify feature vectors such that feature vectors of object detections associated with distinct objects are arranged in distinct clusters in a feature vector space. In this method, a region of the first frame is received corresponding to the object detection in the first frame, and a set of feature vectors associated with the region in the first frame comprising two or more feature vectors is identified by the trained convolutional neural network to identify a reference set of feature vectors comprising two or more feature vectors. A plurality of regions of the second frame are further received corresponding to each object detection of the plurality of object detections in the second frame. For each of the plurality of regions in the second frame, a set of feature vectors associated with that region in the second frame is identified by the trained convolutional neural network to identify a plurality of candidate sets of feature vectors. For each feature vector of the reference set of feature vectors, a predetermined number of feature vectors having a shortest distance in feature vector space to that feature vector of the reference set of feature vectors are identified among the feature vectors of the multiple candidate sets of feature vectors to identify a closest set of feature vectors. For each candidate set of feature vectors of the multiple candidate sets of feature vectors, a measure of closeness of the feature vector to the reference set is determined as a proportion of the closest set of feature vectors that belong to that candidate set of feature vectors. For each of the multiple object detections in the second frame, a weight is assigned based on the determined measure of closeness of the feature vector of the candidate set of feature vectors corresponding to that object detection to the reference set. The object detection in the first frame is associated with one of the multiple object detections in the second frame based on the assigned weight.

[0008] As used herein, the same object refers to the same instance of an object, i.e., the same physical object.

[0009] Object detections related to distinct objects refer to object detections that are not related to the same instance of an object, i.e., related to different physical objects. For example, there may be object detections related to different instances of the same object type. In a situation where there are objects of several different object types, there may also be object detections related to objects of different types.

[0010] Placing feature vectors of object detections related to distinct objects in distinct clusters in feature vector space means that the internal distance in feature vector space between feature vectors of the same object is usually smaller than the distance in feature vector space between feature vectors of different objects. However, outliers may occur, i.e., a feature vector of an object may have a substantially larger distance in vector space to other feature vectors of the object than the internal distance between the other feature vectors of the object. In such cases, the feature vector may have a smaller distance to one or more feature vectors of other objects than it has to other feature vectors of related objects.

[0011] After identifying the closeness measure for each candidate set, a weight corresponding to each object detection is assigned based on the identified closeness measure for the candidate set corresponding to the corresponding object detection according to the method of the first aspect, thereby realizing an effective way of setting weights indicative of similarity between the object detection in the first frame and each object detection in the second frame. Such weights may be used, for example, to add similarity of object appearance properties in relation to associating the object detection in the first frame with one of the object detections in the second frame, thus enhancing the algorithm when such association is made on another basis, such as based on the predicted state of the object in the second frame. Also, since the weights are based on the closeness measure and further take into account all feature vectors for the object detection in the first frame, spatial variation of the feature vectors for the object detection in the first frame is taken into account in the assigned weights.

[0012] By selecting a larger predetermined number of feature vectors with the shortest distance in the feature vector space, the influence of a few outliers of the candidate set on the closeness measure will be reduced so that the closeness measure will be dominated by feature vectors of the respective candidate sets that are located closer to the center of the candidate set in the feature vector space. On the other hand, the predetermined number of feature vectors with the shortest distance in the feature vector space shall not be too high so as to be close to the total number of feature vectors of all candidate sets.

[0013] The assigned weights may be such that for any two object detections of the multiple object detections in the second frame that have the same closeness measure, the assigned weights are the same for the two object detections, and for any object detection of the multiple object detections in the second frame that has a larger closeness measure than another object detection of the multiple object detections in the second frame, the assigned weights are larger for that object detection than the assigned weights for the other object detections. This allows the weights to affect associations when the closeness measures are different, but not when the closeness measures are the same. This is advantageous, for example, when the weights are used to adapt an algorithm in which associations are made on other criteria, such as based on the predicted state of the objects in the second frame. The weights can then be applied to add similarity of the appearance properties of the objects in the association, such that when the weights indicate that the object detection in the second frame has a larger closeness measure, the association is more likely to be made for that object. On the other hand, when the weights of two different objects are the same, the weights do not affect the association, and the association is based only on other criteria, such as the predicted state.

[0014] The assigned weights may be such that for each of the multiple object detections in the second frame, the assigned weights are proportional to a determined measure of closeness of the feature vectors of the candidate set corresponding to that object detection to the reference set of feature vectors. This is advantageous when applying algorithms in which associations are made on another basis, such as based on the predicted state of the objects in the second frame, using the weights. The weights can then be applied to add similarities in the appearance properties of the objects in the associations, such that the weights will affect the associations in proportion to their size.

[0015] The assigned weights may be such that, for each of the multiple object detections in the second frame, the assigned weight is 1 plus the difference between the closeness measure of the object detection and the average closeness measure of the object detections of the multiple object detections in the second frame other than that object detection.

[0016] For each object detection of the multiple object detections in the second frame, an association score indicating the probability that that object detection in the second frame is associated with the object detection in the first frame may be calculated based on the weight for that object detection.

[0017] The association score may further be based on a predicted state in the second frame of the object detection in the first frame.

[0018] The predicted state may include one or more of a predicted size, a predicted shape, a predicted position, a predicted velocity, and a predicted acceleration.

[0019] An object detection in a first frame may be associated with the object detection of the multiple object detections in a second frame that has the highest association score.

[0020] An object detection in a first frame may be associated with a track, and the track may be updated based on an object detection in a second frame with which the object detection in the first frame is associated. By associating an object detection in a first frame with one of the object detections in the second frame based on an assigned weight, the similarity of the object appearance properties may be added, and the algorithm may be enhanced to make such association based on another criteria, such as a predicted state of the object in the second frame. Thus, updating the track based on such association reduces the risk of erroneously updating the track based on an erroneous object detection in the second frame, and thus the risk of the track being lost or starting to follow another object.

[0021] Identifying a set of feature vectors associated with a region in the first frame by the trained convolutional neural network may include identifying a first plurality of feature vectors for a plurality of sub-regions of the first image frame by the trained convolutional neural network, and identifying a reference set of feature vectors including two or more feature vectors by identifying a set of feature vectors of the first plurality of feature vectors associated with a region in the first frame that includes two or more feature vectors. Identifying a set of feature vectors associated with the region in the second frame by the trained convolutional neural network for each of a plurality of regions in the second frame may include identifying a second plurality of feature vectors for a plurality of sub-regions of the second image frame by the trained convolutional neural network, and identifying a plurality of candidate sets of feature vectors by identifying a set of feature vectors associated with the region in the second frame for each of a plurality of regions in the second frame.

[0022] According to a second aspect, there is provided a non-transitory computer readable storage medium having stored thereon instructions for performing the method according to the first aspect when executed by a device having processing capability.

[0023] Where applicable, the above mentioned features of the method according to the first aspect also apply to this second aspect, reference being made thereto in order to avoid excessive repetition.

[0024] According to a third aspect, there is provided a device for correlating object detections in a first frame with object detections in a second frame using a convolutional neural network trained to identify feature vectors such that feature vectors of object detections associated with distinct objects are located in distinct clusters in feature vector space, the device including circuitry configured to perform the method according to the first aspect.

[0025] The above-described features of the method according to the first aspect also apply to this third aspect, where applicable. In order to avoid excessive repetition, reference is made to the above.

[0026] Further scope of applicability of the present invention will become apparent from the detailed description set forth hereinafter. It should be understood, however, that the detailed description and specific examples, while indicating preferred embodiments of the invention, are given by way of illustration only, since various changes and modifications within the scope of the invention will become apparent to those skilled in the art from this detailed description.

[0027] Therefore, it is to be understood that the invention is not limited to the specific components of the device described, or the operation of the method described, since such devices and methods may vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting. It should be noted that, as used in this specification and the appended claims, the articles "a," "an," "the," and "said" are intended to mean that there are one or more of an element, unless the context clearly dictates otherwise. Thus, for example, reference to "a unit" or "the unit" may include several devices, etc. Furthermore, the words "comprising," "including," "containing," and similar phrases do not exclude other elements or steps.

[0028] The above and other aspects of the present invention will now be described in more detail with reference to the accompanying figures, which should not be considered limiting, but rather are used for purposes of explanation and understanding. [Brief description of the drawings]

[0029] [Figure 1] 1 is a flowchart associated with an embodiment of a method of the present disclosure for correlating an object detection in a first frame with an object detection in a second frame using a convolutional neural network. [Figure 2a] FIG. 2 illustrates an example of a first frame including an object. [Figure 2b] FIG. 11 is a diagram illustrating an example of a second frame including multiple objects. [Diagram 3] 2a and 2b show simplified representations of feature vectors corresponding to an object in a first frame shown in FIG. 2a and multiple objects in a second frame shown in FIG. 2b in two-dimensional space. [Figure 4]FIG. 1 is a schematic diagram related to an embodiment of a device of the present disclosure for correlating an object detection in a first frame with an object detection in a second frame using a convolutional neural network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] The present invention will now be described with reference to the accompanying drawings in which presently preferred embodiments of the invention are shown, however, this invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0031] The present invention is applicable in scenarios where an object detection in a first frame is associated with an object detection in a second frame. Such scenarios are for example when tracking an object in a sequence of frames. In such tracking, a track associated with an object identification in a first frame of the sequence of frames shall be updated in relation to a second subsequent frame that contains multiple object detections. In such scenarios, it is important that the object detection in the first frame is associated with exactly one of the multiple object detections in the second frame.

[0032] An embodiment of a method for correlating object detection in a first frame with object detection in a second frame using a convolutional neural network will now be described in relation to FIG. 1 showing a flowchart 100, as well as FIGS. 2a and 2b showing an illustration of a first frame 200 including an object 210 and a second frame 220 including multiple objects, namely a first object 230, a second object 240 and a third object 250, and FIG. 3 showing a simplified representation of feature vectors corresponding in two-dimensional space to the object in the first frame 200 shown in FIG. 2a and the multiple objects in the second frame shown in FIG. 2b.

[0033] The steps of the method illustrated in relation to Figure 1 may be performed for a series of frames. Thus, the method may have been previously performed in relation to an earlier frame as the first frame, and the first frame 200 of Figure 2a as the second frame. Similarly, the method may be subsequently performed in relation to the second frame 220 of Figure 2b as the first frame, and a further subsequent frame as the second frame.

[0034] The convolutional neural network is trained to identify feature vectors such that feature vectors of object detections associated with distinct objects are placed in distinct clusters in the feature vector space. Object detections associated with distinct objects refer to object detections that are not associated with the same instance of an object. For example, distinct objects may be different instances of the same object type or different types of objects.

[0035] A feature vector may be determined for a frame by first determining one feature vector for each of a plurality of sub-regions of the frame, where the plurality of sub-regions preferably together form a complete frame. For example, a frame may be divided into a grid of sub-regions of n×n pixels, where n is a positive non-zero integer. For example, one feature vector may be determined for each pixel in the frame, but typically the spatial resolution of the feature vector is lower, e.g., one feature vector per 8×8 pixels. After the feature vector for the entire frame is determined, the feature vector for the object detection of the frame may then be determined by identifying the feature vector of the sub-region of the frame that is within the region corresponding to the object detection.

[0036] Training may be performed by any known training method that achieves the desired property of the convolutional neural network, i.e., feature vectors of object detections related to distinct objects are placed in distinct clusters in the feature vector space. Such training may be based, for example, on a loss function that encourages the convolutional neural network to map each subregion of a frame to a feature vector in the feature vector space such that subregions belonging to the same object (i.e., the same instance of an object type) are mapped to feature vectors that are close together in the feature vector space, while subregions belonging to different objects (i.e., different instances of the same object type or different object types) are mapped to feature vectors that are far apart in the feature vector space. The distance between the feature vectors may be defined, for example, as the Euclidean distance in the feature vector space. Such a loss function may include a sum over all pairs of each subregion of several training frames. Each term of the sum may, for example, include a first sub-term for pairs of sub-regions belonging to the same object, the first sub-term providing a small contribution of low-value distances between corresponding feature vectors and a large contribution of high-value distances between corresponding mapped feature vectors, and a second sub-term for pairs of sub-regions belonging to different objects, the second sub-term providing a small contribution of high-value distances between corresponding feature vectors and a large contribution of low-value distances between corresponding feature vectors. Each term of the sum may further be normalized with respect to the total number of pixels in the corresponding training image and weighted by the weight of the corresponding pixel and pixel pair for the training image.An example of a neural network that may be used, and how it is trained, is provided in "Recurrent Pixel Embedding for Instance Grouping," Kong, Shu et al., Department of Computer Science, University of California, USA, December 22, 2017 (arXiv:1712.08273v1). Another example of a neural network that may be used is provided in "Semantic Instance Segmentation with a Discriminative Loss Function," De Brabandere et al., ESAT-PSI, KU Leuven, Belgium, August 8, 2017 (arXiv:1708.02551v1). A further example is provided in "Semantic Instance Segmentation via Deep Metric Learning," Fathi et al., Google Inc. and UCLA, USA, March 30, 2017 (arXiv:1703.10277v1). For the purposes of this application, the COCO dataset (http: / / cocodataset.org / #home) may be used for this training.

[0037] For embodiments in which there are objects of different object types, the convolutional neural network needs to be trained using images that include objects of different object types, such that pixels that belong to objects of different object types are mapped to feature vectors that are far apart in the feature vector space. This can be achieved, for example, by training any of the neural networks mentioned above using the COCO dataset.

[0038] In this method, a region of a first frame is received S110 corresponding to an object detection in the first frame. For example, in the first frame 200 shown in FIG. 2a, an object 210 is detected. The region corresponding to the object detection is a region of the object 210 in the first frame 200 having a particular size, shape, and location. The object detection may be identified, for example, by an object instance segmentation algorithm that provides information about different object detections and their location in the image. The information about the location may be in the form of a mask that can be used to mask out the region corresponding to the object detection. One example of an object instance segmentation algorithm that can be used is Mask R-CNN, described in the paper "Mask R-CNN" by He et al. from January 14, 2018 (arXiv:17.03.06870v3). The paper describes the Mask R-CNN framework, how it is trained, and suitable training data.

[0039] A reference set of feature vectors is identified S120 as a set of feature vectors associated with the receiving region in the first frame. For this purpose, the trained convolutional neural network described above may be used. A complete set of feature vectors may be identified for the first frame by identifying one feature vector for each sub-region of a plurality of sub-regions of the frame, where the plurality of sub-regions preferably together form a complete frame. A reference set of feature vectors may then be identified by identification of feature vectors of the complete set of feature vectors associated with sub-regions located in the receiving region of the first frame. Alternatively, it is sufficient that only feature vectors associated with the receiving region are identified for the first frame. A reference set of feature vectors may then be identified by identifying feature vectors for sub-regions located within the receiving region of the first frame. In relation to the object 210 in the first frame 200 shown in FIG. 2a, a reference set of feature vectors is identified as a set of feature vectors for sub-regions located within the region of the object 210.

[0040] The reference set of feature vectors includes two or more feature vectors. This can be achieved by selecting the size of the sub-regions in which the corresponding feature vectors are identified so that the region corresponding to the object detection includes at least two sub-regions. Alternatively or additionally, only objects of at least a minimum size related to the size of the sub-regions are allowed for association so that each object has at least a certain number of feature vectors. This is advantageous because objects need to be of a reasonable size so that association (re-identification) can be performed successfully. Furthermore, the more feature vectors there are for an object, the more spatial variation of the object can be taken into account and the more statistical basis for association. The certain number of feature vectors can be two, but is usually more than this.

[0041] A plurality of regions of the second frame are further received S130 corresponding to each of the plurality of object detections in the second frame. For example, objects 230, 240, 250 are detected in the second frame 220 shown in FIG. 2b. The plurality of regions corresponding to the plurality of object detections are regions of objects 230, 240, 250 having a particular size, shape, and position in the second frame 220. These object detections may also be identified using an object segmentation algorithm such as Mask-RCNN described above.

[0042] A plurality of candidate sets of feature vectors are then identified S140 as a plurality of sets of feature vectors associated with a plurality of received regions in the second frame. For this purpose, a trained convolutional neural network may be used. As described in relation to the first frame, a complete set of feature vectors may also be identified for the second frame. A candidate set of feature vectors may then be identified for each object detection of the plurality of object detections of the second frame of the complete set of feature vectors by identification of a feature vector associated with a sub-region located within the region corresponding to that object detection. Alternatively, it is sufficient that only the feature vectors associated with the plurality of received regions are identified for the second frame. A candidate set of feature vectors may then be identified for each region of the plurality of received regions by identifying a feature vector for a sub-region located within that received region. In relation to the objects 230, 240, 250 in the second frame 220 shown in FIG. 2b, a plurality of candidate sets of feature vectors are identified for each of the plurality of objects 230, 240, 250 as a set of feature vectors for sub-regions located within the region of that object.

[0043] The set of closest feature vectors is then identified S150 by identifying, for each feature vector of the reference set of feature vectors, a predetermined number of feature vectors in the multiple candidate sets of feature vectors that have the shortest distance in feature vector space to this feature vector of the reference set. For this purpose, the same distance metric used when training the convolutional neural network can be used, although there may be other choices. Thus, for each feature vector of the reference set, a distance in feature vector space is determined for each feature vector of the multiple candidate sets of feature vectors, and the predetermined number of feature vectors of the multiple candidate sets of feature vectors that have the shortest distance are included in the set of closest feature vectors.

[0044] The predetermined number of feature vectors is preferably equal to or less than the number of feature vectors of the candidate set having the smallest number of feature vectors. For example, if a certain lower limit is set on the number of feature vectors of each object, the predetermined number can be set to be less than the certain lower limit. This is advantageous because then all feature vectors of a single object are likely to be closest to the respective vectors of the reference set in the feature vector space.

[0045] The predetermined number of feature vectors can be set to 1, but it may be advantageous to set it to a larger number, since then the influence of outliers, i.e. feature vectors of the reference and candidate sets that are located at a relatively long distance in the feature vector space away from other feature vectors of the same reference or candidate set, is reduced. The predetermined number of feature vectors may, for example, be set to a certain percentage of the number of feature vectors of the candidate set that has a minimum number of feature vectors.

[0046] FIG. 3 shows a simplified representation of feature vectors corresponding to an object 210 in a first frame 200 shown in FIG. 2a and multiple objects 230, 240, 250 in a second frame 220 shown in FIG. 2b in a two-dimensional space. The feature vectors are represented as different geometric shapes of points in a two-dimensional space. For the object 210 in the first frame 200 shown in FIG. 2a, a reference set of feature vectors consisting of four feature vectors has been identified. In FIG. 3, the four feature vectors consisting of a first feature vector 312, a second feature vector 314, a third feature vector 316, and a fourth feature vector 318 of the reference set are represented as four circles. For the first object 230 in the second frame 220 shown in FIG. 2b, a first candidate set of feature vectors consisting of three feature vectors has been identified. In Fig. 3, the three feature vectors of the first candidate set, consisting of the first feature vector 332, the second feature vector 334, the third feature vector 336, and the fourth feature vector 338, are represented as four plus signs. For the second object 240 in the second frame 220 shown in Fig. 2b, a second candidate set of feature vectors consisting of the four feature vectors has been identified. In Fig. 3, the four feature vectors of the second candidate set, consisting of the first feature vector 342, the second feature vector 344, the third feature vector 346, and the fourth feature vector 348, are represented as four squares. For the third object 250 in the second frame 220 shown in Fig. 2b, a second candidate set of feature vectors consisting of the four feature vectors has been identified. In Fig. 3, the four feature vectors of the second candidate set, consisting of a first feature vector 352, a second feature vector 354, a third feature vector 356, and a fourth feature vector 358, are represented as four triangles. The number of feature vectors for the reference set and each of the candidate sets is all four. This is for illustrative purposes only. Typically, the number of feature vectors may be 10 or 100 or more. Furthermore, the number of feature vectors of the reference set may be different from the number of each of the feature vectors of the candidate sets, and the number of feature vectors may vary among the candidate sets.For a given number of feature vectors with a shortest distance equal to 4, and a distance that is the Euclidean distance in two-dimensional space, the feature vectors with shortest distance to each feature vector of the reference set are as follows: For a first feature vector 312 of the reference set, the four closest feature vectors of the candidate set are the first feature vector 332 of the first candidate set, and the first feature vector 342, the second feature vector 344, and the third feature vector 346 of the second candidate set. For a second feature vector 314 of the reference set, the four closest feature vectors of the candidate set are the first feature vector 342 and the second feature vector 344 of the second candidate set, and the first feature vector 352 and the second feature vector 354 of the third candidate set. For the third feature vector 316 of the reference set, the four closest feature vectors of the candidate set are the first feature vector 332 of the first candidate set, the first feature vector 342 and the second feature vector 344 of the second candidate set, and the first feature vector 352 of the third candidate set. For the fourth feature vector 318 of the reference set, the four closest feature vectors of the candidate set are the first feature vector 342, the second feature vector 344, the third feature vector 346, and the fourth feature vector 348 of the second candidate set, as summarized in Table 1 below. TIFF0007660489000001.tif77170

[0047] Returning to Figure 1, a measure of closeness of the feature vector to a reference set is determined for each candidate set of feature vectors of a plurality of candidate sets of feature vectors S160. The measure of closeness is determined as the proportion of the set of closest feature vectors that belong to that candidate set of feature vectors.

[0048] As can be seen in Table 1, the total number of feature vectors in the set of closest feature vectors of the candidate set is 16. For the first candidate set, the first feature vector 332 is included twice in the set of closest feature vectors. Hence, the closeness measure of the first candidate set is 2 / 16. For the second candidate set, in the set of closest feature vectors, the first feature vector 342 and the second feature vector 344 are both included four times, the third feature vector 346 is included twice, and the fourth feature vector is included once. Thus, the closeness measure of the second candidate set is 11 / 16. For the third candidate set, in the set of closest feature vectors, the first feature vector 352 is included twice, and the second feature vector 354 is included once. Thus, the closeness measure of the third candidate set is 3 / 16.

[0049] A weight is assigned S170 to each of the multiple object detections in the second frame, with each weight being based on a determined closeness measure of the feature vector of the candidate set corresponding to that object detection S170 to the reference set of feature vectors. Thus, with reference to FIG. 2b, the first weight assigned to the first object 230 is based on the determined closeness measure of the first candidate set, the second weight assigned to the second object 240 is based on the determined closeness measure of the second candidate set, and the third weight assigned to the third object 250 is based on the determined closeness measure of the third candidate set.

[0050] Because the assigned weights are based on a measure of closeness of each candidate set corresponding to the object detection in the second frame to the reference set corresponding to the object detection in the first frame, and the manner in which the closeness measures are determined, the assigned weights will typically indicate a measure of similarity in appearance between the object in the first frame and the object in the second frame. The weights may also be referred to as appearance-based weights.

[0051] By selecting a higher predetermined number of feature vectors with the shortest distance in the feature vector space, the influence of a few outliers in the candidate set on the closeness measure will be reduced such that the closeness measure will be dominated by feature vectors that are located closer to the center of the candidate set in the feature vector space.

[0052] The assigned weights are preferably such that if two object detections of the plurality of object detections in the second frame have the same closeness measure, then the weight assigned to each of them is the same, and if a first object detection has a higher closeness measure than a second object detection, then the assigned weight is higher for the first object detection than for the second object detection. Furthermore, the weights assigned to the object detections may be proportional to the determined closeness measure of the corresponding candidate set.

[0053] The object detection in the first frame may then be associated S180 with one of the object detections in the second frame based on the assigned weight. For example, the assigned weight may be used to calculate S175 an association score for each combination of the one of the object detections in the second frame and the object detection in the first frame. Each association score indicates a probability that the associated object detection in the second frame and the object detection in the first frame are associated with the same object and based on the weight for the associated object detection in the second frame in relation to the object detection in the first frame. The object detection in the first frame may then be associated S180 with the object detection of the multiple object detections in the second frame that has the highest association score.

[0054] For example, if the weights and association scores are based solely on the similarity of appearance of each of the multiple objects in the second frame to the object in the first frame, the weight for each of the multiple object detections in the second frame may be set equal to a measure of the closeness of the object detection to the object detection in the first frame. The association score for each of the multiple object detections in the second frame relative to the object detection in the first frame can then be set equal to the weight of that object detection.

[0055] For objects 230, 240, 250 in the second frame 220 shown in FIG. 2b, and the closeness measures determined based on Table 1, the weight and association score for the first object 230 is then 2 / 16. The weight and association score for the second object 240 is 11 / 16. The weight and association score for the third object 250 is 3 / 16.

[0056] Each of the association scores may further be based on a predicted state in the second frame of the object detection in the first frame, e.g., based on Kalman filter state variables. The predicted state may include one or more of a predicted size, a predicted shape, a predicted position, a predicted velocity, and an acceleration. The predicted state in the second frame of the object detection in the first frame is typically based on the track to which the object detection in the first frame is associated. The track is updated with the identified state in each frame after association to the object detection. It should be noted that the identified state after association does not have to be identical to the characteristics of the associated object detection. For example, the identified state may differ from that of the associated object detection in terms of one or more of size, shape, position, velocity, and acceleration. The predicted state in the second frame of the object detection in the first frame can be compared to each of the object detections in the second frame to identify a probability that the object detection in the first frame and the associated object detection in the second frame are associated with the same object. For example, the distance d j may be determined as the difference between the predicted state in the second frame of the object detection in the first frame and the j-th object detection in the second frame. j may simply be the distance between the predicted state in the second frame of the object detection in the first frame and the center point of the object based on the jth object detection in the second frame, or may be based on the overlap of the object based on the predicted state in the second frame of the object detection in the first frame and the jth object detection in the second frame, e.g., as measured by Intersection over Union.

[0057] Distance d jmay be converted to a probability that an object detection in the first frame corresponds to the jth object detection in the second frame. For example, a Gaussian curve or other curve that decays with increasing distance may be used to convert distance to probability. As an example, the probability P j teeth, The pixel size may be calculated according to TIFF0007660489000002.tif12170, where σ is a predefined constant.

[0058] Distance d j is the appearance-based weight w, which is calculated based on the closeness measure. j For example, the distances may be divided by the weights and possibly scaled by a constant. Based on such distance modification, the modified probability or association score may be TIFF0007660489000003.tif14170, where α is a constant that dampens the influence of the appearance-based weights.

[0059] When using appearance-based weights to modify the probabilities based on the predicted states, the assigned weights may be such that, for each of a plurality of object detections in the second frame, the assigned weight is 1 plus the difference between the closeness measure of that object detection and the average closeness measure of the object detections of the plurality of object detections in the second frame other than that object detection. Thus, if there are m object detections in the second frame and the closeness measure of the jth object detection in the second frame to the object detection in the first frame is a j Then, the weights for the jth object detection, w j teeth, Calculated according to TIFF0007660489000004.tif11170.

[0060] For the objects 230, 240, 250 in the second frame 220 shown in FIG. 2b and the closeness measure determined based on Table 1, the weight w1 for the first object 230 according to Equation 2 is 1+(2 / 16-1 / 2 * The weight w2 for the second object 240 according to Equation 1 is 1+(11 / 16-1 / 2 * The weight w3 for the third object 250 according to Equation 1 is 1+(3 / 16-1 / 2 * (2 / 16+11 / 16))=25 / 32.

[0061] The object detection in the first frame may then be associated S180 with the object detection of the multiple object detections in the second frame that has the highest revised probability, i.e., the highest association score.

[0062] The association may be used, for example, in connection with updating a track associated with an object detection in a first frame based on multiple object detections in a second frame. The track may then be updated S185 based on an object detection in the second frame with which the object detection in the first frame is associated.

[0063] The above describes how one object detection in a first frame is associated with one object detection among multiple object detections in a second frame. However, the method of the present disclosure is also applicable to the case where multiple object detections in a first frame are associated with multiple object detections in a second frame. In such a scenario, a respective weight is assigned to each object detection of the multiple object detections in the second frame in relation to each object detection of the multiple object detections in the first frame.

[0064] The multiple object detections in the first frame may then be associated with each of the multiple object detections in the second frame based on the assigned weight S180. For example, the assigned weight may be used to calculate S175 an association score for each combination of one of the multiple object detections in the second frame with one of the multiple object detections in the first frame. Each association score indicates a probability based on the weight for the associated object detection in the second frame that the associated object detection in the first frame and the associated object detection in the second frame are associated with the same object and are associated with the associated object detection in the first frame. The association score may then be used in an algorithm to associate each of the multiple object detections in the first frame with at most one of the multiple object detections in the second frame, such that each of the object detections in the second frame is associated with at most one of the multiple object detections in the first frame. For example, when using the Hungarian algorithm to associate each object detection in a first frame (the previous frame) with at most one object detection of multiple object detections in a second frame (the current frame), the input to the Hungarian algorithm is, for each detection of multiple detections in the second frame, the probability that that object detection in the first frame is associated with that object detection in the second frame, i.e., how likely they are to be the same object. Hence, an association score may be calculated as such a probability.For example, if the weights and association scores are based solely on the similarity of appearance between each of the multiple objects in the second frame and each of the multiple objects in the first frame, the weight for each of the multiple object detections in the second frame associated with each of the multiple object detections in the first frame may be set equal to a corresponding closeness measure for each of the multiple object detections in the second frame associated with each of the multiple object detections in the first frame. The association score for each of the multiple object detections in the second frame associated with each of the multiple object detections in the first frame can then be set equal to a corresponding weight for each of the multiple object detections in the second frame associated with each of the multiple object detections in the first frame.

[0065] Furthermore, the association algorithm includes a threshold below which an object detection in a first frame is not associated with any object detection in a second frame. In such a case, the track to which the object detection in the first frame is associated is considered to lose tracking. The tracking can then be terminated, or a timeout can be used where the track is updated based only on the predicted state without using any associated object detection in the second frame to see if association can be made in subsequent frames. The algorithm may also further support the creation of a corresponding new track in association with an object detection in the second frame that is not associated with any object detection in the first frame. To avoid creating a new track based on an object detection in the second frame that appears in only one or a few frames and is due to noise, a delay may be used such that a new track is created only if the object detection is always made for more than a certain number of frames.

[0066] Each of the association scores may further be based on a predicted state in the second frame of a corresponding object detection of the plurality of object detections in the first frame, for example, based on Kalman filter state variables. The predicted state may include one or more of a predicted size, a predicted shape, a predicted position, a predicted velocity, and an acceleration. The predicted state in the second frame of a corresponding object detection of the plurality of object detections in the first frame is typically based on a track to which the corresponding object detection of the plurality of object detections in the first frame is associated. The track is updated with the identified state in the respective frame after association to the object detection. It should be noted that the identified state after association does not have to be identical to the characteristics of the associated object detection. For example, the identified state may differ from that of the associated object detection in terms of one or more of size, shape, position, velocity, and acceleration. For each object detection of the plurality of object detections in the first frame, the predicted state in the second frame of that object detection in the first frame can be compared to the object detection in the second frame to determine a probability that the object detection in the first frame and the related object detection in the second frame relate to the same object. For example, a distance d kj may be determined as the difference between the predicted state in the second frame of the kth object detection in the first frame and the jth object detection in the second frame. kj may simply be the distance between the predicted state in the second frame of the kth object detection in the first frame and the center point of the object based on the jth object detection in the second frame, or may be based on the overlap of the object based on the predicted state in the second frame of the kth object detection in the first frame and the jth object detection in the second frame, for example, as measured by Intersection over Union.

[0067] Distance d kj may be converted to the probability that the kth object detection in the first frame corresponds to the jth object detection in the second frame. For example, a Gaussian curve or other curve that decays with increasing distance may be used to convert distance to probability. As an example, the probability P kj teeth, The pixel size may be calculated according to TIFF0007660489000005.tif14170, where σ is a predefined constant.

[0068] Distance d kj is the appearance-based weight w, which is calculated based on the closeness measure. kj For example, the distances may be divided by the weights and possibly scaled by a constant. Based on such distance modification, the modified probability or association score may be TIFF0007660489000006.tif15170, where α is a constant that dampens the influence of the appearance-based weights.

[0069] When using appearance-based weights to modify probabilities based on predicted states, the assigned weights for multiple object detections in a first frame may each be calculated in the same manner as described in relation to the assigned weights for one object detection in the first frame. For example, if there are m object detections in a second frame, and the measure of closeness of the jth object detection in the second frame to the kth object detection in the first frame is a kj If we denote the weight w for the kth object detection in the first frame relative to the jth object detection in the second frame, kj teeth, Calculated according to TIFF0007660489000007.tif11170.

[0070] Each object detection of the multiple object detections in the first frame may then be associated with the object detection of the multiple object detections in the second frame that has the highest association score S180. Alternatively, the association score may then be used in an algorithm to associate each object detection of the multiple object detections in the first frame with at most one object detection of the multiple object detections in the second frame, such that each object detection of the object detections in the second frame is associated with at most one object detection of the multiple object detections in the first frame. For example, if the Hungarian algorithm is used, an association score may be determined based on the assigned weights and the predicted states, thereby indicating how likely it is that the kth object detection in the first frame is associated with the jth object detection of the multiple object detections in the second frame, i.e., the probability P that they are the same object, e.g., according to Equation 5: kj Make sure to show the following.

[0071] The association may be used, for example, in connection with updating tracks associated with multiple object detections in a first frame based on multiple object detections in a second frame, For each object detection of the multiple object detections in the first frame, a corresponding track may then be updated based on an object detection in the second frame with which that object detection in the first frame is associated S185.

[0072] 4 shows a schematic diagram associated with an embodiment of a device 400 of the present disclosure for correlating an object detection in a first frame with an object detection in a second frame using a convolutional neural network. The device 400 includes a circuit 410. The circuit 410 is configured to perform the functions of the device 400. The circuit 410 may include a processor 412, such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 412 is configured to execute program code. The program code may be configured to perform the functions of the device 400, for example.

[0073] The device 400 may further include a memory 430. The memory 430 may be one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, a random access memory (RAM), or another suitable device. In a typical arrangement, the memory 430 includes a non-volatile memory for long-term data storage and a volatile memory that serves as a system memory for the circuit 410. The memory 430 may exchange data with the circuit 410 over a data bus. There may also be associated control lines and an address bus between the memory 430 and the circuit 410.

[0074] The functionality of the image processing device 400 may be embodied in the form of executable logic routines (e.g. lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (e.g. memory 430) of the device 400 and executed by the circuit 400 (e.g. using the processor 412). Furthermore, the functionality of the device 400 may be a standalone software application or may form part of a software application that performs further tasks related to the device 400. The described functionality may be considered as a method that a processing unit, e.g. the processor 412 of the circuit 410, is configured to execute. Also, although the described functionality may be implemented in software, such functionality may be performed by dedicated hardware or firmware, or any combination of hardware, firmware, and / or software.

[0075] The circuit 410 is configured to perform a first region receiving function 431 configured to receive a region of the first frame corresponding to an object detection in the first frame, and a reference set identifying function 432 configured to identify a reference set of feature vectors comprising two or more feature vectors by identifying, via a trained convolutional neural network, a set of feature vectors associated with a region in the first frame comprising the two or more feature vectors.

[0076] The circuit 410 is further configured to perform a second region receiving function 433 configured to receive a plurality of regions in the second frame corresponding to each of the plurality of object detections in the second frame, and a candidate set identification function 434 configured to identify, for each of the plurality of regions in the second frame, a plurality of candidate sets of feature vectors by identifying, by the trained convolutional neural network, a set of feature vectors associated with that region in the second frame.

[0077] The circuit 410 is further configured to perform a nearest feature vector identification function 435 configured to identify, for each feature vector of the reference set of feature vectors, a set of nearest feature vectors by identifying, among the feature vectors of the plurality of candidate sets of feature vectors, a predetermined number of feature vectors that have a shortest distance in feature vector space to that feature vector of the reference set of feature vectors.

[0078] The circuit 410 is further configured to execute a closeness measure determination function 436 configured to determine, for each candidate set of feature vectors of a plurality of candidate sets of feature vectors, a closeness measure to a reference set of feature vectors as a proportion of the set of closest feature vectors that belong to that candidate set of feature vectors.

[0079] The circuit 410 is further configured to execute a weight assignment function 437 configured to assign a weight for each of a plurality of object detections in the second frame based on an identified measure of closeness of the feature vector of the candidate set corresponding to that object detection to a reference set of feature vectors.

[0080] The circuit 410 is further configured to perform an association function 439 configured to associate the object detection in the first frame with one of the multiple object detections in the second frame based on the assigned weights.

[0081] The weight assignment function 437 may be configured to assign weights such that for any two object detections of the multiple object detections in the second frame having the same closeness measure, the assigned weights are the same for the two object detections, and such that for any object detection of the multiple object detections in the second frame having a higher closeness measure than another object detection of the multiple object detections in the second frame, the assigned weight for that object detection is higher than the assigned weight for that other object detection.

[0082] The circuit 410 is further configured to execute an association score calculation function 438 configured to calculate, for each object detection of the plurality of object detections in the second frame, an association score indicative of a probability that the object detection in the second frame is associated with the object detection in the first frame based on a weight for that object detection. The association score may be further based on a predicted state in the second frame of the object detection in the first frame. The predicted state may include one or more of a predicted size, a predicted shape, a predicted position, a predicted velocity, and an acceleration.

[0083] The association function 439 may be configured to associate an object detection in a first frame with an object detection of the multiple object detections in a second frame that has the highest association score.

[0084] The object detection in the first frame may be associated with a track, and the circuitry may be further configured to perform a track update function 440 configured to update the track based on an object detection in a second frame to which the object detection in the first frame is associated.

[0085] The functions performed by the circuit 410 may further be adapted as corresponding steps of the method embodiments described in relation to FIGS. 1, 2a and 2b, and FIG.

[0086] It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims. Such modifications and variations can be understood and effected by those skilled in the art in practicing the claimed invention, by studying the drawings, the disclosure, and the appended claims. [Explanation of symbols]

[0087] 200 1st Frame 220 Second Frame 210 Objects 230 First Object 240 Second Object 250 Third Object 312 First feature vector 314 Second feature vector 316 Third Feature Vector 318 Fourth Feature Vector 332 First feature vector 334 Second feature vector 336 Third Feature Vector 338 Fourth Feature Vector 342 First feature vector 344 Second feature vector 346 Third Feature Vector 348 Fourth Feature Vector 352 First feature vector 354 Second feature vector 356 Third Feature Vector 358 Fourth Feature Vector 400 devices 410 Circuit 412 processor 430 Memory 431 First Area Reception Function 432 Criteria Set Identification Function 433 Second Area Reception Function 434 Candidate set identification function 435 Recent Feature Vector Classification Function 436 Closeness Measure Specific Functions 437 Weight assignment function 438 Association score calculator 439 Association Function 440 Track Update Function

Claims

1. 1. A method for correlating object detections in a first frame with object detections in a second frame using a convolutional neural network trained to identify feature vectors such that feature vectors of object detections associated with distinct objects are located in distinct clusters in a feature vector space, comprising: receiving a region of the first frame corresponding to the object detection in the first frame; identifying a reference set of feature vectors comprising two or more feature vectors by identifying a set of feature vectors associated with the region in the first frame comprising two or more feature vectors using the trained convolutional neural network; receiving a plurality of regions of the second frame corresponding to each object detection of a plurality of object detections in the second frame; for each of the plurality of regions in the second frame, identifying a plurality of candidate sets of feature vectors by identifying a set of feature vectors associated with the region in the second frame by the trained convolutional neural network; identifying, for each feature vector of the reference set of feature vectors, a set of closest feature vectors among the feature vectors of a plurality of candidate sets of feature vectors by identifying a predetermined number of feature vectors having a shortest distance in the feature vector space to the feature vectors of the reference set of feature vectors; determining, for each candidate set of feature vectors of the plurality of candidate sets of feature vectors, a measure of closeness of the feature vector to a reference set as a proportion of the closest set of feature vectors that belong to the candidate set of feature vectors; assigning, for each of the plurality of object detections in the second frame, a weight based on the determined measure of closeness of the candidate set of feature vectors corresponding to the object detection to the reference set of feature vectors; and associating the object detection in the first frame with one of the plurality of object detections in the second frame based on the assigned weights.

2. for any two object detections among the plurality of object detections in the second frame having the same closeness measure, the assigned weights are the same for the two object detections; and 2. The method of claim 1 , wherein for any object detection of the plurality of object detections in the second frame that has a greater closeness measure than another object detection of the plurality of object detections in the second frame, the assigned weight is greater for the any object detection than the assigned weight for the other object detection.

3. 3. The method of claim 1 or 2, wherein for each of the plurality of object detections in the second frame, the assigned weight is proportional to the determined measure of closeness of the candidate set of feature vectors corresponding to the object detection to the reference set of feature vectors.

4. 2. The method of claim 1 , wherein, for each of the plurality of object detections in the second frame, the assigned weight is one plus a difference between the closeness measure of the object detection and an average closeness measure of the object detections of the plurality of object detections in the second frame other than the object detection.

5. 5. The method of claim 1 , further comprising: for each object detection of the plurality of object detections in the second frame, calculating an association score indicating a probability that the object detection in the second frame is associated with the object detection in the first frame based on the weight for the object detection.

6. The method of claim 5 , wherein the association score is further based on a predicted state in the second frame of the object detection in the first frame.

7. The method of claim 6 , wherein the predicted state includes one or more of a predicted size, a predicted shape, a predicted position, a predicted velocity, and an acceleration.

8. The method of claim 1 , wherein the object detection in the first frame is associated with the object detection of the plurality of object detections in the second frame that has the highest association score.

9. 9. The method of claim 1 , further comprising: updating the track based on the object detection in the second frame to which the object detection in the first frame is associated.

10. The act of identifying, by the trained convolutional neural network, a set of feature vectors associated with the region in the first frame, comprises: identifying a first plurality of feature vectors for a plurality of sub-regions of the first image frame by the trained convolutional neural network; identifying a set of feature vectors of the first plurality of feature vectors associated with the region in the first frame that includes two or more feature vectors, thereby determining a reference set of feature vectors that includes two or more feature vectors; For each of the plurality of regions in the second frame, the operation of identifying, by the trained convolutional neural network, a set of feature vectors associated with the region in the second frame includes identifying, by the trained convolutional neural network, a second plurality of feature vectors for a plurality of sub-regions of the second image frame; and for each of the plurality of regions in the second frame, identifying a set of feature vectors associated with the region in the second frame, thereby identifying a plurality of candidate sets of feature vectors.

11. A non-transitory computer readable storage medium having stored thereon instructions for performing the method of any one of claims 1 to 10 when executed by a device having processing capability.

12. 1. A device for correlating object detections in a first frame with object detections in a second frame using a convolutional neural network trained to identify feature vectors such that feature vectors of object detections associated with distinct objects are located in distinct clusters in a feature vector space, comprising: receiving a region of the first frame corresponding to the object detection in the first frame; identifying a reference set of feature vectors comprising two or more feature vectors by identifying, with the trained convolutional neural network, a set of feature vectors associated with the region in the first frame comprising two or more feature vectors; receiving a plurality of regions of the second frame corresponding to each object detection of a plurality of object detections in the second frame; for each of the plurality of regions in the second frame, identifying a plurality of candidate sets of feature vectors by identifying a set of feature vectors associated with the region in the second frame with the trained convolutional neural network; identifying, for each feature vector of the reference set of feature vectors, a set of closest feature vectors among the feature vectors of a plurality of candidate sets of feature vectors by identifying a predetermined number of feature vectors having a shortest distance in the feature vector space to the feature vectors of the reference set of feature vectors; determining, for each candidate set of feature vectors of the plurality of candidate sets of feature vectors, a measure of closeness of the feature vector to a reference set as a proportion of the closest set of feature vectors that belong to the candidate set of feature vectors; assigning, for each of the plurality of object detections in the second frame, a weight based on the determined measure of closeness of the candidate set of feature vectors corresponding to the object detection to the reference set of feature vectors; and a first frame including a first iteration of the plurality of object detections and a second frame including a second iteration of the plurality of object detections.

13. 13. The device of claim 12, wherein for any two object detections of the multiple object detections in the second frame having a same closeness measure, the assigned weights are the same for the two object detections; and, for any object detection of the multiple object detections in the second frame that has a greater closeness measure than another object detection of the multiple object detections in the second frame, the assigned weight is greater for the any object detection than the assigned weight for the other object detection.

14. 14. The device of claim 12 or 13, wherein the circuitry is further configured to calculate, for each object detection of the plurality of object detections in the second frame, an association score indicating a probability that the object detection in the second frame is associated with the object detection in the first frame based on the weight for the object detection.

15. The device of claim 14 , wherein the association score is further based on a predicted state in the second frame of the object detection in the first frame.

Images