Devices and methods for enhancing object tracking in a scene

Abstract

Devices and methods for enhancing object tracking in a scene are disclosed. Specifically, devices, media, and methods for selecting a clipping score threshold for enhancing object tracking in a scene are disclosed. A corresponding trajectory for each object among different objects is obtained, the trajectory including a clipping, each clipping having a clipping score and a feature vector. Each trajectory is segmented into trajectory segments. For each candidate clipping score threshold: the difference between each trajectory segment and each other trajectory segment is determined based on the difference between the feature vector of the clipping with a clipping score higher than the candidate clipping score threshold for each of four or more trajectory segments and the feature vector of the clipping with a clipping score higher than the candidate clipping score threshold for each of four or more other trajectory segments. From a set of candidate clipping score thresholds, a clipping score threshold is selected that results in the largest difference between the difference between trajectory segments of different trajectories and the difference between trajectory segments of the same trajectories.

Description

Technical Field

[0001] This invention relates to object tracking, and more particularly to selecting a cropping score threshold for enhancing tracking of objects in a scene captured in a video sequence. Background Technology

[0002] In existing technology, when tracking objects in a scene captured in a video sequence, trajectory segments are identified, comprising clipped subsets from subsequent image frames of the video sequence. Each clipped segment is a portion of an image frame in the video sequence and is associated with a detection in that image frame. For example, each clipped segment could be a portion of a corresponding image frame within a bounding box surrounding an object detected in an image frame. All clipped segments of the trajectory have been determined to involve the same object according to given criteria. The trajectory segments belonging to the same object are then concatenated to generate a single trajectory involving that same object. To determine whether two different trajectory segments belong to the same tracked object, feature vectors of the clipped segments of the corresponding trajectory segments can be used. Such feature vectors can be, for example, feature vectors determined by a convolutional neural network. Specifically, the feature vector of each of the two trajectory segments can be determined, for example, as the average of the feature vectors of the clipped segments of the corresponding trajectory segments, and if the distance between their feature vectors is less than a threshold, the trajectory segments can be determined to belong to the same tracked object. A potential problem with this existing approach for connecting trajectory segments is that the feature vectors of the determined trajectory segments may not represent objects, and therefore errors may occur, such as determining that two trajectory segments do not involve the same object even if they actually do. Summary of the Invention

[0003] The purpose of this invention is to provide a method and apparatus for selecting a cropping score threshold for enhancing tracking of objects in a scene captured in a video sequence, which overcomes or mitigates problems in the prior art.

[0004] This disclosure utilizes the understanding that the clipping score of each clipping can be used to determine which clippings of a trajectory segment should be used to determine the difference between that trajectory segment and other trajectory segments.

[0005] According to a first aspect, a method is provided for selecting a cropping score threshold for enhancing tracking of objects in a scene captured in a video sequence. A corresponding trajectory for each of two different objects is obtained, wherein each trajectory includes a crop of an object instance of the corresponding object in the scene captured in the calibration video sequence, and wherein each crop has a cropping score and a feature vector. Each trajectory is then segmented into two or more corresponding trajectory segments, resulting in four or more trajectory segments. For each candidate cropping score threshold in a set of candidate cropping score thresholds: a corresponding difference between each of the four or more trajectory segments and each of the other four or more trajectory segments is determined based on the difference between the feature vector of the crop with a cropping score higher than the cropping score threshold of each of the four or more trajectory segments and the feature vector of the crop with a cropping score higher than the cropping score threshold of each of the four or more other trajectory segments. The cropping score threshold that results in the largest difference between a measure of the determined difference between trajectory segments of different trajectories and a measure of the determined difference between trajectory segments of the same trajectories is then selected from the set of candidate cropping score thresholds.

[0006] By selecting a clipping score threshold that maximizes the difference between the determined difference metric between trajectory segments of different trajectories and the determined difference metric between trajectory segments of the same trajectories, a more accurate distinction can be achieved between trajectory segments of different trajectories and trajectory segments from the same trajectories. Specifically, a difference threshold can be set for the determined difference between trajectory segments, where it is likely that the determined difference between trajectory segments of different trajectories is greater than the difference threshold, and the determined difference between trajectory segments of the same trajectories is less than the difference threshold.

[0007] "Cropping" refers to a region of an image frame in a video sequence. Typically, cropping involves removing pixels that primarily depict the object. Cropping can, for example, correspond to a region enclosed by a bounding box. Alternatively, cropping can correspond to a rectangular image region, where a segmented mask obscures the detected object, and pixels within the rectangular image region that do not belong to the detected object are set to zero.

[0008] The term "track segment" refers to a portion of the trajectory of an object that relates to a subset of sequential image frames in a video sequence. The combination of all track segments relating to the same object in the video sequence constitutes the trajectory of that object.

[0009] A "feature vector" is generally an n-dimensional vector representing the numerical features of an object. This is a common representation of objects used in, for example, machine learning algorithms, which typically require numerical representations of objects for purposes such as data processing and statistical analysis. Feature vectors can be received, for example, from machine learning modules trained to extract similar feature vectors for the same object in different image frames and less similar feature vectors for different objects.

[0010] The action of determining the corresponding difference may include determining the corresponding feature vector for each of four or more trajectory segments based on the clipped feature vectors of the trajectory segments having clipping scores higher than the candidate clipping score thresholds, and determining the corresponding difference between each of the four or more trajectory segments and each of the other four or more trajectory segments as the corresponding difference between each determined corresponding feature vector and each other determined corresponding feature vector. In the action of selecting a clipping score threshold from a set of candidate clipping score thresholds, the clipping score threshold that results in the largest difference between the measure of the determined difference between feature vectors from different trajectories and the measure of the determined difference between feature vectors from the same trajectory segment is then selected.

[0011] In the action of determining the corresponding feature vector, the corresponding feature vector of each of the four or more trajectory segments can be determined as the average or median of the feature vectors of the trajectory segment that have a clipping score higher than the candidate clipping score threshold.

[0012] In the action of selecting the cropping score threshold, the metric can be the average or the median.

[0013] The method according to the first aspect may further include identifying a subset of candidate cropping score thresholds, consisting of each candidate cropping score threshold among candidate cropping score thresholds, wherein for each candidate cropping score threshold, the number of croppings having cropping scores higher than the candidate cropping score threshold is greater than or equal to a first desired number for each of four or more trajectory segments. In the action of selecting a cropping score threshold, a cropping score threshold is then selected from the subset of candidate cropping score thresholds.

[0014] By selecting a cropping score threshold from a subset of candidate cropping score thresholds, each trajectory segment that results in the identification of objects in the scene will include at least one cropping score that is more likely to be cropped than the selected cropping score.

[0015] The method according to the first aspect may further include acquiring two further trajectory segments involving one or more objects, wherein each trajectory segment includes a crop of an object instance of one or more objects in a scene captured in a video sequence, and wherein each crop has a crop score and a feature vector. Then, based on the feature vector of the cropped segment having a crop score higher than a selected crop score threshold, a corresponding feature vector for each of the two further trajectory segments is determined, and then a difference is determined between the determined corresponding feature vectors of the two further trajectory segments. The determined difference is then used to determine whether the two further trajectory segments involve the same object or different objects. For example, if the determined difference is lower than a difference threshold, the two further trajectory segments may be determined to involve the same object, and if the determined difference is higher than a difference threshold, the two further trajectory segments may be determined to involve different objects. In the action of determining the corresponding feature vector for each of the two further trajectory segments, if the number of clippings of the trajectory segment with clipping scores higher than the selected clipping score threshold is less than the second desired number, the corresponding feature vector can be further determined based on another clipping with the maximum clipping score among the clippings of the trajectory segment with clipping scores equal to or lower than the selected clipping score threshold, such that the number of clippings on which the feature vector is based is equal to the second desired number.

[0016] According to a second aspect, a non-transitory computer-readable storage medium is provided, on which instructions are stored for implementing the method according to the first aspect or the method according to the first aspect when executed by a device having processing capabilities.

[0017] Where applicable, the optional features mentioned above in the method of the first aspect also apply to the non-transitory computer-readable storage medium of the second aspect. To avoid unnecessary repetition, refer to the foregoing.

[0018] According to a third aspect, an apparatus is provided for enhancing the tracking of objects in a scene captured in a video sequence. The apparatus includes circuitry configured to perform a first acquisition function, a segmentation function, a first determination function, and a selection function. The first acquisition function is configured to acquire a corresponding trajectory for each of two different objects, wherein each trajectory includes a crop of an object instance of the corresponding object in the scene captured in the calibration video sequence, and wherein each crop has a crop score and a feature vector determined by a classifier. The segmentation function is configured to segment each trajectory into corresponding two or more trajectory segments, thereby forming four or more trajectory segments. The first determination function is configured to: for each candidate crop score threshold in a set of candidate crop score thresholds, determine a corresponding difference between each of the four or more trajectory segments and each of the other four or more trajectory segments based on the difference between the feature vector of the crop with a crop score higher than the candidate crop score threshold of each of the four or more trajectory segments and the feature vector of the crop with a crop score higher than the candidate crop score threshold of each of the four or more trajectory segments. The selection function is configured to select from a set of candidate clipping score thresholds the clipping score threshold that results in the largest difference between the measure of the determined difference between trajectory segments of different trajectories and the measure of the determined difference between trajectory segments of the same trajectory.

[0019] Where applicable, the optional features mentioned above regarding the method of the first aspect also apply to the device of the third aspect. To avoid unnecessary repetition, refer to the above.

[0020] According to a fourth aspect, a method for enhancing the tracking of objects in a scene captured in a video sequence is provided. Trajectories involving one or more objects in the scene captured in the video sequence are obtained, wherein the trajectory includes cropped instances of object instances of the objects in the one or more objects, and wherein each cropped instance has a cropping score and a feature re-identification vector. A feature vector of the trajectory is determined based on the feature vector of the cropped instance having a cropping score higher than a cropping score threshold selected for the scene. A further cropped instance of one of the object instances in the scene captured in the video sequence is obtained, wherein the cropped instance has a cropping score and a feature vector selected for the scene. If the cropping score of the further cropped instance is greater than the cropping score threshold selected for the scene, the difference between the determined feature vector of the trajectory and the feature vector of the further cropped instance is then determined. Based on the determined difference, it is determined whether the further cropped instance and the trajectory involve the same object or different objects. The selected cropping score threshold can be selected, for example, according to the method according to the first aspect.

[0021] If a further crop has a crop score greater than the crop score threshold selected for the scene, then false identification of further cropping involving the same object can be avoided by only determining the difference and whether the further cropping involves the same object as the trajectory. Furthermore, processing associated with other croppings having crop scores equal to or less than the crop score threshold can be avoided.

[0022] According to a fifth aspect, an apparatus is provided for enhancing the tracking of objects in a scene captured in a video sequence. The apparatus includes circuitry configured to perform a first acquisition function, a first determination function, a second acquisition function, and a second determination function. The first acquisition function is configured to acquire trajectories relating to one or more objects in the scene captured in the video sequence, wherein the trajectory includes clippings of object instances of the objects in the one or more objects, and wherein each clipping has a clipping score and a feature re-identification vector. The first determination function is configured to determine a feature vector of the trajectory based on the feature vector of a clipping segment having a clipping score higher than a selected clipping score threshold. The second acquisition function is configured to acquire a further clipping of an object instance of one of the one or more objects in the scene captured in the video sequence, wherein the clipping has a clipping score and a feature vector selected for the scene. If the clipping score of the further clipping is greater than the selected clipping score threshold for the scene, the second determination function is configured to determine the difference between the determined corresponding feature vectors of the two trajectory segments, and determine, based on the determined difference, whether the further clipping and trajectory relate to the same object or different objects. The clipping score threshold selected for the scene can be chosen, for example, according to the method in the first aspect.

[0023] The further scope of the invention will become apparent from the detailed description given below. However, it should be understood that while indicating preferred embodiments of the invention, the detailed description and specific examples are given only by way of illustration, as various variations and modifications within the scope of the invention will become apparent to those skilled in the art from this detailed description.

[0024] Therefore, it should be understood that the present invention is not limited to the specific components of the described apparatus or the operation of the described method, as such apparatus and methods can vary. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It must be noted that, unless the context clearly indicates otherwise, the terms “a,” “an,” “the,” and “described” as used in the specification and appended claims are intended to indicate the presence of one or more elements. Furthermore, the words “comprising,” “containing,” “including,” and similar terms do not exclude other elements or steps. Attached Figure Description

[0025] The above and other aspects of the invention will now be described in more detail with reference to the accompanying drawings. The drawings should not be considered limiting, but rather for explanation and understanding.

[0026] Figure 1a and Figure 1b A flowchart relating to an embodiment of the method of this disclosure is shown.

[0027] Figure 2 A schematic diagram relating to an embodiment of the device disclosed herein is shown.

[0028] Figure 3 A flowchart relating to an embodiment of another method of this disclosure is shown.

[0029] Figure 4 A schematic diagram relating to an embodiment of another device of this disclosure is shown. Detailed Implementation

[0030] The invention will now be described below with reference to the accompanying drawings, which illustrate presently preferred embodiments of the invention. However, the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0031] This invention is applicable to scenarios where objects are tracked within a scene captured over time in a video sequence. This may occur, for example, when a scene is captured in a sequence of image frames by a surveillance camera. Object detection modules using any type of object detection have been used to detect objects. Feature vectors involving cropping of the tracked object in each image frame can be extracted during object detection, or feature vectors can be extracted in a separate process using object detection as input.

[0032] In the following, reference will be made to Figure 1a and Figure 1b Examples of a method 100 for selecting a cropping score threshold to enhance tracking of objects in a scene captured in a video sequence are discussed.

[0033] Obtain the corresponding trajectory for each of the two different objects in S110. Each trajectory includes a crop of the object instance of the corresponding object in the scene captured in the calibration video sequence. Each crop has a crop score and a feature vector. In addition to the crop included in each trajectory, each trajectory may further include an estimate associated with some frames where no crop is available, for example because no object was detected in these frames. Such a crop score can be determined by annotating the crop based on whether its contribution to the problem-solving is positive or negative, and then training a classifier on that data.

[0034] The objects may involve vehicles, people, faces, bags, pets, or any other objects that may be of interest in different situations.

[0035] A cropping score can be, for example, a value indicating how well the cropping works for re-identification, or in other words, how well the cropping performs as input for extracting feature vectors for object re-identification. Alternatively, a cropping score can be a value indicating how well the cropping is for object classification, or in other words, how well the cropping is as input for extracting feature vectors for object classification. This cropping score can be determined by a classifier trained to compute the value of the cropping (i.e., the cropping score), which specifies how similar the cropping is to what is considered a “good” cropping. For example, the cropping score can be a value between 0 and 1.

[0036] The cropped feature vector is a vector describing the features of an object. Feature vectors can be received, for example, from machine learning modules such as neural networks trained to extract similar feature vectors of the same object in different image frames and less similar feature vectors of different objects. Alternatively, feature vectors can be received from other types of modules, such as those based on support vector machines or color histograms. The way the cropping score is determined will depend on the type of module from which the feature vectors are received.

[0037] Each trajectory is then segmented S120 into two or more corresponding trajectory segments, resulting in four or more trajectory segments. It should be noted that the two received trajectories associated with the calibration video sequence are known to relate to one of two different objects. Therefore, it is known which of the four or more trajectory segments belongs to which trajectory. For example, this can be achieved by selecting calibration video sequences with well-separated trajectories from two different and simultaneously tracked objects in the scene.

[0038] For each candidate clipping score threshold in a set of candidate clipping score thresholds, determine the corresponding difference between each of the four or more trajectory segments in S130 and each of the other four or more trajectory segments. Each corresponding difference is based on the difference between a feature vector of clipped segments with clipping scores higher than the candidate clipping score thresholds of the four or more trajectory segments and a feature vector of clipped segments with clipping scores higher than the candidate clipping score thresholds of another trajectory segment. Thus, a first difference is determined based on the difference between a feature vector of clipped segments with clipping scores higher than the candidate clipping score thresholds of a first trajectory segment and a feature vector of clipped segments with clipping scores higher than the candidate clipping score thresholds of a second trajectory segment, a second difference is determined based on the difference between a feature vector of clipped segments with clipping scores higher than the candidate clipping score thresholds of a first trajectory segment and a feature vector of clipped segments with clipping scores higher than the candidate clipping score thresholds of a third trajectory segment, and so on, until a difference has been determined for each combination of four or more trajectory segments. Therefore, for n trajectory segments, n*(n-1) / 2 differences will be determined for each candidate clipping score. The difference between two feature vectors can be determined, for example, by determining the distance between the two feature vectors. The larger the distance between the two feature vectors, the larger the difference. The distance can be, for example, in terms of L... 2 The norm (also known as the Euclidean norm) is used to determine this.

[0039] Preferably, the difference between two trajectory segments should be determined such that the greater the difference between the feature vector of a clipped segment with a clipping score higher than the candidate clipping score threshold of one of the two trajectory segments and the feature vector of a clipped segment with a clipping score higher than the candidate clipping score threshold of the other of the two trajectory segments, the greater the difference between the two trajectory segments. For example, the action of determining the corresponding difference in S130 may include determining the corresponding feature vector for each of four or more trajectory segments based on the feature vector of the clipped segment with a clipping score higher than the candidate clipping score threshold. Preferably, the feature vector of the trajectory segment is determined such that it is, for example, a reflection of the mean or median of the feature vector of the clipped segment with a clipping score higher than the candidate clipping score threshold of the trajectory segment. The corresponding difference between each of the four or more trajectory segments and each of the other four or more trajectory segments can then be determined as the corresponding difference between each determined corresponding feature vector and each of the other determined corresponding feature vectors. In addition to determining the corresponding feature vector for each trajectory segment, the variance of the feature vectors may also be determined. In this case, the corresponding difference between each of the four or more trajectory segments and each of the other four or more trajectory segments can then be determined as the corresponding difference between each determined corresponding eigenvector and corresponding variance, with or without weighting, and each of the other determined corresponding eigenvectors and corresponding variances, with or without weighting.

[0040] Furthermore, when the tracked object in a trajectory segment has more than one appearance within that segment, more than one feature vector can be determined for each trajectory segment. For example, if the tracked object is a person, two feature vectors can be determined: a first feature vector relating to the clipping of the person visible from the front, and a second feature vector relating to the clipping of the person visible from the rear. The first and second feature vectors can then be determined using the mean or median feature vector of the clipping of the person visible from the front and the second feature vector of the clipping of the person visible from the rear, respectively. When the difference between the first and second trajectory segments each has more than one feature vector, the difference can be determined as a measure of the corresponding difference between the corresponding feature vectors of the first and second trajectory segments. This measure can be, for example, the median or mean.

[0041] The candidate cropping scores for a set of candidate cropping score thresholds will vary depending on the type of cropping score used. For example, if a cropping score that can have values ​​in the range of 0 to 1 is used, the candidate cropping scores could be 0.1, 0.2, ..., 0.9. Of course, other ranges and other values ​​within those ranges are also possible.

[0042] Then, from a set of candidate clipping score thresholds, a clipping score threshold is selected that results in the maximum difference between a measure of the determined difference between trajectory segments of different trajectories and a measure of the determined difference between trajectory segments of the same trajectories, as determined by S140. The measure is preferably such that it reflects, respectively, the determined difference between trajectory segments of different trajectories and the determined difference between trajectory segments of the same trajectories, such as the average or median.

[0043] Furthermore, a clipping score threshold is preferably selected such that the measure of the difference between trajectory segments of the same trajectory is smaller than the measure of the difference between trajectory segments of different trajectories.

[0044] In an embodiment where the action of determining the corresponding difference in S130 includes determining a corresponding feature vector for each of four or more trajectory segments, and the corresponding difference between each trajectory segment and each other trajectory segment is determined as the corresponding difference between each determined corresponding feature vector and each other determined corresponding feature vector, a clipping score threshold is selected as the maximum difference between the measure of the determined difference between feature vectors of trajectory segments of different trajectories and the measure of the determined difference between feature vectors of trajectory segments of the same trajectories. The measure is preferably such that it reflects the determined difference between feature vectors of trajectory segments of different trajectories and the determined difference between feature vectors of trajectory segments of the same trajectories, such as an average or median. Then, more preferably, a clipping score threshold is selected such that the measure of the determined difference between feature vectors of trajectory segments of the same trajectories is less than the measure of the determined difference between feature vectors of trajectory segments of different trajectories.

[0045] The selected clipping score threshold can then be used to determine which clippings of the trajectory to use when determining the feature vector of the trajectory, which is used to compare with the feature vector of a further clipping or another trajectory when determining whether the compared trajectories involve the same or different objects.

[0046] The selected clipping score threshold is typically used for the scenario already selected by method 100 and associated with it. For different scenarios, a new clipping score threshold is typically determined by performing method 100 associated with the different scenarios.

[0047] The cropping score threshold should be high enough so that the cropping used for re-identification is "good" enough. However, if the cropping score threshold is too high, it is possible that no cropping will be higher than the threshold for short trajectories. Method 100 may therefore further include identifying, in S135, a subset of candidate cropping score thresholds consisting of each candidate cropping score threshold among the candidate cropping score thresholds, wherein for each candidate cropping score threshold, the number of croppings having a cropping score higher than the candidate cropping score threshold is greater than or equal to a first desired number for each of four or more trajectory segments. In the action of selecting a cropping score threshold, a cropping score threshold is then selected from the subset of candidate cropping score thresholds. By selecting a cropping score threshold from the subset of candidate cropping score thresholds, a cropping score threshold that is more likely to result in each trajectory associated with an object in the scene being included in the identification of the object is selected.

[0048] Once a cropping score threshold is selected, the selected cropping threshold can be used to determine whether two further trajectories involve the same object or different objects. Method 100 can therefore further include obtaining, in S150, two further trajectories involving one or more objects. Each of the two further trajectories includes a crop of an object instance of one or more objects in a scene captured in the video sequence, and each crop has a cropping score and a feature vector. Then, based on the feature vector of the crop with a cropping score higher than the selected cropping score threshold from the action selected in S140, the corresponding feature vector of each of the two further trajectories in S160 is determined. Then, in S170, the difference between the determined corresponding feature vectors of the two further trajectories is determined. The difference between the two feature vectors can be determined, for example, by determining the distance between the two feature vectors. The greater the distance between the two feature vectors, the greater the difference. The distance can be, for example, in the form of L... 2 The norm (also known as the Euclidean norm) is used to determine this.

[0049] Then, based on the determined difference, it is determined whether the two further trajectories in S180 involve the same object or different objects. For example, if the determined difference is below a difference threshold, the two further trajectories can be determined to involve the same object, and if the determined difference is above a difference threshold, the two further trajectories can be determined to involve different objects. For the selected clipping score threshold for selecting the action in S140, the difference threshold can be set based on the measure of the determined difference between the trajectories of different trajectories and the measure of the determined difference between the trajectories of the same trajectories. Specifically, for the selected clipping score threshold for selecting the action in S140, the difference threshold can be set to the average of the measure of the determined difference between the trajectories of different trajectories and the measure of the determined difference between the trajectories of the same trajectories. For example, the measure can be the average or the median. If, for example, the measure of the determined difference between the trajectories of different trajectories is 1.2, and the measure of the determined difference between the trajectories of the same trajectories is 0.2, then the difference threshold can be set to 0.7.

[0050] To ensure that at least a second required number of clippings are used for each further trajectory to determine the corresponding feature vector for each of the two further trajectories, if the number of clippings for a trajectory with a clipping score higher than the selected clipping score threshold is less than the second required number, the corresponding feature vector can be further determined based on an additional clipping of the trajectory with the largest clipping score among the clippings with clipping scores equal to or lower than the selected clipping score threshold, such that the number of clippings on which the feature vector is based is equal to the second required number. Therefore, if a trajectory in the two further trajectories has fewer clippings than the second required number and has a clipping score higher than the selected clipping score threshold, an additional clipping is used to determine the feature vector of the trajectory. Alternatively, a further clipping score threshold can be set, which is a lower clipping score, wherein clipping is not used to determine the feature vector even if this results in fewer clippings than the second required number used to determine the feature vector.

[0051] In the following, reference will be made to Figure 2 An embodiment of a device 200 for selecting a cropping score threshold to enhance tracking of objects in a scene captured in a video sequence is discussed.

[0052] Device 200 includes circuitry 210. Circuitry 210 is configured to perform the functions of device 200. Circuitry 210 may include processor 212, such as a central processing unit (CPU), microcontroller, or microprocessor. Processor 212 is configured to execute program code. The program code may, for example, be configured to perform the functions of device 200.

[0053] Device 200 may further include memory 230. Memory 230 may be one or more of a buffer, flash memory, hard disk drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or another suitable device. In a typical arrangement, memory 230 may include non-volatile memory for long-term data storage and volatile memory serving as device memory for circuit 210. Memory 230 may exchange data with circuit 210 via a data bus. Accompanying control lines and address buses may also exist between memory 230 and circuit 210.

[0054] The functionality of device 200 can be embodied in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory 230) of device 200 and executed by circuitry 210 (e.g., using processor 212). Furthermore, the functionality of device 200 can be a standalone software application or part of a software application that performs additional tasks related to device 200. The described functionality can be considered as a method configured to be executed by a processing unit (e.g., processor 212 of circuitry 210). Similarly, while the described functionality can be implemented in software, it can also be implemented via dedicated hardware or firmware, or a combination of hardware, firmware, and / or software.

[0055] Circuit 210 is configured to perform a first acquisition function 231, a segmentation function 232, a first determination function 233, and a selection function 234.

[0056] The first acquisition function 231 is configured to acquire the corresponding trajectory of each of two different objects, wherein each trajectory includes a crop of the object instance of the corresponding object in the scene captured in the calibration video sequence, and wherein each crop has a crop score and feature vector determined by a classifier.

[0057] The segmentation function 232 is configured to divide each trajectory into two or more corresponding trajectory segments, thereby forming four or more trajectory segments.

[0058] For each candidate clipping score threshold in a set of candidate clipping score thresholds, the first determining function 233 is configured to: determine the corresponding difference between each of the four or more trajectory segments and each of the four or more other trajectory segments based on the difference between the feature vector of clipping with clipping scores of candidate clipping score thresholds higher than each of the four or more trajectory segments and the feature vector of clipping with clipping scores of candidate clipping score thresholds higher than each of the four or more other trajectory segments.

[0059] Selection function 234 is configured to select from a set of candidate clipping score thresholds the clipping score threshold that results in the largest difference between the measure of the determined difference between trajectory segments of different trajectories and the measure of the determined difference between trajectory segments of the same trajectory.

[0060] Circuit 210 may be further configured to perform identification function 235, second acquisition function 236, second determination function 237, third determination function 238 and fourth determination function 239.

[0061] Recognition function 235 is configured to identify a subset of candidate cropping score thresholds for which the number of croppings having cropping scores higher than the candidate cropping score thresholds is greater than or equal to a first desired number for each of four or more trajectory segments. In selection function 234, a cropping score threshold is then selected from the subset of candidate cropping score thresholds.

[0062] In the first determination function 233, the corresponding feature vector of each of the four or more trajectory segments can be determined as the average or median of the feature vectors of the trajectory segments that have a clipping score higher than the candidate clipping score threshold.

[0063] In option 234, the measure can be the average or the median.

[0064] The second acquisition function 236 is configured to acquire two further trajectories involving one or more objects, wherein each trajectory includes a crop of an object instance of one or more objects in a scene captured in a video sequence, and wherein each crop has a crop score and feature vector determined by a classifier.

[0065] The second determination function 237 is configured to determine the corresponding feature vector of each of the two further trajectories based on the feature vector of the trajectory with a clipping score higher than the selected clipping score threshold.

[0066] The third determination function 238 is configured to determine the difference between the determined corresponding feature vectors of two further trajectories.

[0067] The fourth determination function 239 is configured to determine, based on the determined difference, whether two further trajectories involve the same object or different objects.

[0068] The fourth determination function 239 can be configured to determine that two further trajectories involve the same object if the determined difference is below a difference threshold, and to determine that two further trajectories involve different objects if the determined difference is above a difference threshold.

[0069] In the second determination function 237, if the number of clippings of a trajectory with a clipping score higher than the selected clipping score threshold is less than the second required number, then a corresponding re-identification vector is further determined based on another clipping of the trajectory with the largest clipping score among the clippings with clipping scores equal to or lower than the selected clipping score threshold, such that the number of clippings on which the re-identification vector is based is equal to the second required number.

[0070] Further details regarding the functions that circuit 210 is configured to perform are provided in relation to the corresponding actions of the method 100 for selecting a cropping score threshold for enhancing tracking of objects in a scene captured in a video sequence.

[0071] In the following, the combination Figure 3 Examples of a method 300 for enhancing the tracking of objects in a scene captured in a video sequence are discussed.

[0072] In the following, reference will be made to Figure 3 Examples of a method 300 for enhancing the tracking of objects in a scene captured in a video sequence are discussed.

[0073] Obtaining S350 involves capturing trajectories of objects within one or more objects in a scene captured in a video sequence. The trajectory comprises cropped instances of the objects within the one or more objects. Each cropped instance has a crop score and a feature vector. This can be described as follows: Figure 1a The description determines the cutting score.

[0074] Based on the feature vector of the corresponding trajectory, which has a cropping score higher than the cropping score threshold selected for the scene, an S360 feature vector is determined for the trajectory. This can be as follows: Figure 1a The description is used to determine the feature vector.

[0075] Further cropping is performed on an object instance of one or more objects captured in a scene by S365 in a video sequence. The further cropping has a cropping score and feature vector selected for the scene. This can be done as described above. Figure 1a The description determines the cutting score.

[0076] Under condition C368, where the cropping score of the further cropping is greater than the cropping score threshold selected for the scene, the difference between the feature vector of the trajectory determined in S370 and the feature vector of the further cropping is determined. Then, based on the determined difference, it is determined whether the further cropping and trajectory in S380 involve the same object or different objects. This can be discussed as follows... Figure 1a The description is used to determine the difference.

[0077] Under condition C368, where the cropping score of further cropping is not greater than the cropping score threshold, further cropping is not considered in comparison with the trajectory, because this indicates that the cropping is insufficient to represent the object, making feature vector comparison unlikely to lead to relevant results.

[0078] It should be noted that in other embodiments, the comparison is made between two tracks. Here, the comparison is made between a single clipping and a track. For this purpose, a single clipping may be a special case where the track consists of only one clipping (i.e., a further clipping).

[0079] According to about Figure 1a The described method 100 is used to select the clipping score threshold for the scene.

[0080] In the following, reference will be made to Figure 4 An embodiment of a device 400 for enhancing the tracking of objects in a scene captured in a video sequence is discussed.

[0081] Device 400 includes circuitry 410. Circuitry 410 is configured to perform the functions of device 400. Circuitry 410 may include processor 412, such as a central processing unit (CPU), microcontroller, or microprocessor. Processor 412 is configured to execute program code. The program code may, for example, be configured to perform the functions of device 400.

[0082] Device 400 may further include memory 430. Memory 430 may be one or more of a buffer, flash memory, hard disk drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or another suitable device. In a typical arrangement, memory 430 may include non-volatile memory for long-term data storage and volatile memory serving as device memory for circuit 410. Memory 430 may exchange data with circuit 410 via a data bus. Accompanying control lines and address buses may also exist between memory 430 and circuit 410.

[0083] The functionality of device 400 can be embodied in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory 430) of device 400 and executed by circuitry 410 (e.g., using processor 412). Furthermore, the functionality of device 400 can be a standalone software application or part of a software application that performs additional tasks related to device 400. The described functionality can be considered as a method configured to be executed by a processing unit (e.g., processor 412 of circuitry 410). Similarly, while the described functionality can be implemented in software, it can also be implemented via dedicated hardware or firmware, or a combination of hardware, firmware, and / or software.

[0084] Circuit 410 is configured to perform a first acquisition function 436, a first determination function 437, a second acquisition function 438, and a second determination function 439.

[0085] The first acquisition function 436 is configured to acquire trajectories involving one or more objects in a scene captured in a video sequence, wherein the trajectory includes cropping of object instances of the objects in the one or more objects, and wherein each cropping has a cropping score and a feature re-identification vector.

[0086] The first determination function 437 is configured to determine the feature vector of the trajectory based on the feature vector of the trajectory having a clipping score higher than the selected clipping score threshold.

[0087] The second acquisition function 438 is configured to acquire a further crop of an object instance of one or more objects in a scene captured in a video sequence, wherein the crop has a cropping score and feature vector selected for the scene.

[0088] If the cropping score of the further cropping is greater than the cropping score threshold selected for the scene, the second determining function 439 is configured to determine the difference between the determined corresponding feature vectors of the two trajectories, and based on the determined difference, determine whether the further cropping and the trajectory involve the same object or different objects. The cropping score threshold selected for the scene can be selected, for example, according to the method according to the first aspect.

[0089] Further details regarding the functions that circuit 410 is configured to perform are provided in relation to the corresponding actions of method 300 for enhancing the tracking of objects in a scene captured in a video sequence.

[0090] Those skilled in the art will recognize that the present invention is not limited to the embodiments described above. Rather, many modifications and variations are possible within the scope of the appended claims. Such modifications and variations can be understood and implemented by those skilled in the art in practicing the claimed invention through study of the drawings, the disclosure, and the appended claims.

Claims

1. A method for selecting a cropping score threshold for enhancing tracking of objects in a scene captured in a video sequence, the method comprising: Obtain the corresponding trajectory for each of two different objects, wherein each trajectory includes a crop of the object instance of the corresponding object in the scene captured in the calibration video sequence, and wherein each crop has a crop score and a feature vector determined by a classifier, wherein the crop score is a value indicating how well the crop works for re-identification or how well the crop is classified for the object, and wherein the feature vector is a vector describing the features of the corresponding object; Each trajectory is divided into two or more corresponding trajectory segments, thus forming four or more trajectory segments; For each candidate cropping score threshold in a set of candidate cropping score thresholds: Based on the feature vectors of the clipping segments having clipping scores higher than the candidate clipping score threshold, determine the corresponding feature vector for each of the four or more trajectory segments; and The corresponding difference between each of the four or more trajectory segments and each of the other four or more trajectory segments is determined as the corresponding difference between the determined corresponding feature vector of each trajectory segment and the determined corresponding feature vector of each other trajectory segment; and From the set of candidate clipping score thresholds, select the maximum difference between the measure of the determined difference between feature vectors from trajectory segments of different trajectories and the measure of the determined difference between feature vectors from trajectory segments of the same trajectories.

2. The method according to claim 1, wherein, In the action of determining the corresponding feature vector of each of the four or more trajectory segments, the corresponding feature vector of each of the four or more trajectory segments is determined as the average or median of the feature vectors of the trajectory segment that have a clipping score higher than the candidate clipping score threshold.

3. The method according to claim 1, wherein, In the action of selecting the cropping score threshold, the metric is the average or median.

4. The method according to claim 1, further comprising: Identify a subset of candidate cropping score thresholds, consisting of each of the candidate cropping score thresholds, wherein for each subset of candidate cropping score thresholds, the number of croppings having cropping scores higher than the candidate cropping score thresholds is greater than or equal to a first desired number for each of the four or more trajectory segments. Specifically, in the action of selecting a cropping score threshold, the cropping score threshold is selected from a subset of the candidate cropping score thresholds.

5. The method of claim 1, further comprising: Two further trajectory segments involving one or more objects are obtained, wherein each trajectory segment includes a crop of an object instance of one or more objects in the scene captured in the video sequence, and wherein each crop has a crop score and a feature vector; Based on the feature vector of the clipping with a clipping score higher than the selected clipping score threshold of the corresponding trajectory segment, the corresponding feature vector of each of the two further trajectory segments is determined. Determine the difference between the corresponding feature vectors of the two further trajectory segments; and Based on the determined difference, it is determined whether the two further trajectory segments involve the same object or different objects.

6. The method according to claim 5, wherein, Determining whether the two trajectory segments involve the same object or different objects includes: If the determined difference is below a difference threshold, then the two further trajectory segments are determined to involve the same object; and If the determined difference is higher than the difference threshold, it is determined that the two further trajectory segments involve different objects.

7. The method according to any one of claims 5 and 6, wherein, In the action of determining the corresponding feature vector of each of the two further trajectory segments, if the number of clippings of the trajectory segment with clipping scores higher than the selected clipping score threshold is less than the second required number, then the corresponding feature vector of each further trajectory segment is further determined based on another clipping with the maximum clipping score among the clippings of the trajectory segment with clipping scores equal to or lower than the selected clipping score threshold, such that the number of clippings on which the corresponding feature vector of each further trajectory segment is based is equal to the second required number.

8. A non-transitory computer-readable storage medium storing instructions that, when executed by a device having processing capabilities, implement the method according to claim 1.

9. An apparatus for enhancing the tracking of objects in a scene captured in a video sequence, the apparatus comprising circuitry configured to perform the following steps: The first acquisition function is configured to acquire the corresponding trajectory of each of two different objects, where... Each trajectory includes a crop of an object instance of a corresponding object in the scene captured in a calibration video sequence, and wherein each crop has a crop score and a feature vector determined by a classifier, wherein the crop score is a value indicating how well the crop works for re-identification or how well the crop is classified for the object, and wherein the feature vector is a vector describing the features of the corresponding object. The segmentation function is configured to divide each trajectory into two or more corresponding trajectory segments, thereby forming four or more trajectory segments; The first determining function, for each candidate cropping score threshold in a set of candidate cropping score thresholds, is configured to: determine a corresponding feature vector for each of the four or more trajectory segments based on the feature vector of the cropped segment having a cropping score higher than the candidate cropping score threshold; and determine the corresponding difference between each of the four or more trajectory segments and each other of the four or more trajectory segments as the difference between the determined corresponding feature vector of each trajectory segment and the determined corresponding feature vector of each other trajectory segment; and The selection function is configured to select from the set of candidate clipping score thresholds the clipping score threshold that results in the largest difference between the measure of the determined difference between trajectory segments of different trajectories and the measure of the determined difference between trajectory segments of the same trajectories.

10. The device according to claim 9, further comprising: The identification function is configured to identify a subset of candidate cropping score thresholds, wherein for the subset of candidate cropping score thresholds, the number of croppings having cropping scores higher than the candidate cropping score thresholds is greater than or equal to a first desired number for each of the four or more trajectory segments. In the selection function, the cropping score threshold is selected from a subset of the candidate cropping score thresholds.

11. The device according to any one of claims 9 and 10, further comprising: The second acquisition function is configured to acquire two further trajectory segments involving one or more objects, wherein each trajectory segment includes a crop of an object instance of one or more objects in the scene captured in the video sequence, and wherein each crop has a crop score and feature vector determined by a classifier; The second determining function is configured to determine the corresponding feature vector of each of the two further trajectory segments based on the feature vector of the clipped segment having a clipping score higher than a selected clipping score threshold; and The third determining function is configured to determine the difference between the determined corresponding feature vectors of the two further trajectory segments; and The fourth determination function is configured to determine, based on the determined difference, whether the two further trajectory segments involve the same object or different objects.

12. A method for enhancing the tracking of objects in a scene captured in a video sequence, the method comprising: Select a clipping score threshold for the scene according to any one of claims 1 to 7; Obtaining trajectories involving one or more objects in the scene captured in the video sequence, wherein the trajectories include cropping of object instances of the objects in the one or more objects, and wherein each cropping has a cropping score and a feature vector; The feature vector of the trajectory is determined based on the feature vector of the clipping with a clipping score higher than the clipping score threshold selected for the scene. Further cropping of an object instance of one or more objects in the scene captured in the video sequence, wherein the cropping has a cropping score and feature vector selected for the scene; and If the cropping score of the further cropping is greater than the cropping score threshold selected for the scene: determine the difference between the determined feature vector of the trajectory and the feature vector of the further cropping; and determine whether the further cropping and the trajectory involve the same object or different objects based on the determined difference.

Images