Method of determining color of object to be tracked

Abstract

To provide a computer implementation method capable of determining the color of an object to be tracked, a system, and software.SOLUTION: The method includes the steps: determining the color rendering metric for each of multiple areas in a scene by using a first video sequence and detected foreground objects; and when the metric determines that the tracked object has different colors in different images of a second video sequence, selecting the color detected in an area associated with a higher color rendering metric than in an area associated with a lower color rendering metric.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to determining the colour of a tracked object in a video sequence showing a scene, and in particular to determining the colour of a tracked object in a video sequence showing a scene under varying lighting conditions. [Background technology]

[0002] A colour cast refers to the overall tint of a particular colour that pervades an image, usually the result of a particular lighting condition. A variant of this phenomenon, uneven colour cast, refers to different hues that appear in different areas of an image. This can happen when various light sources with different colour temperatures illuminate different sides of a scene. For example, in an outdoor setting, daylight and street lighting may co-exist, creating mixed lighting conditions. Daylight may cast a cooler, bluer light on some parts of a scene, while street lighting may add a warmer, yellowish light on other parts. In addition to this, shadows can also result in localised colour casts. The colour of light in the shadows is often different from direct light, causing an uneven colour cast.

[0003] An example of an application where non-uniform color casts are important to handle is in surveillance systems, especially when an operator inspects video sequences to locate objects of a particular color. Manually combing through multiple video sequences to identify items of interest can be tedious and time-consuming. To alleviate this issue, automated video analysis tools are generally employed to analyze video sequences. These tools can detect objects in the sequence and annotate the detected objects with attributes, such as object type, size, and speed. These features can then be utilized by an operator when searching for specific objects of interest, for example, in forensic search applications. An important feature when searching for objects in an image or video sequence is the color of the object. However, the apparent color of the object will be affected by ambient light sources, such as natural and artificial light sources, which can create a false impression of the color of the object in the image. An example of this is a white car under a yellow street light, which can cause the color of the car in an image capturing the scene to be yellow instead of white.

[0004] A further problem is that lighting conditions in a scene vary for many different reasons, such as the placement of active artificial light sources, time of day, season, weather conditions, architecture, vegetation, etc. This makes it difficult to accurately determine the actual or natural color of an object captured in an image, which can adversely affect the ability to search for an object based on its color.

[0005] US Patent Application No. 2007 / 154088 describes color identification in digital images with the aim of determining the robust perceived color or true color of an object, defined as the color of the object as perceived under several standard viewing, lighting and detection conditions.

[0006] CN107 292 933 describes the use of neural networks to determine color.

[0007] Thierry Bouwmans et al., "On the role and the importance of features for background modelling and foreground detection", Arxiv.org, Cornell University Library, 28 November 2016, XP080735023 describes modelling and chromatic variance for detection purposes.

[0008] There is therefore a need for improvement in this context. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] U.S. Patent Application No. 2007 / 154088 [Patent Document 2] CN107 292 933 [Non-patent literature]

[0010] [Non-Patent Document 1] Thierry Bouwmans et al., "On the role and the importance of features for background modelling and foreground detection," Arxiv.org, Cornell University Library, November 28, 2016, XP080735023 Summary of the Invention

[0011] In view of the above, it would be advantageous to overcome or at least reduce one or some of the above-described disadvantages as set forth in the accompanying independent claims.

[0012] According to a first aspect of the present invention, there is provided a computer-implemented method for determining a colour of a tracked object comprising the steps of: providing a first video sequence showing a scene, the first video sequence comprising a plurality of image frames; for each image frame of the plurality of image frames detecting a foreground object in the image frame; for each area of ​​a plurality of areas in the scene, analysing the first video sequence and calculating an object colour probability vector associated with the area of ​​the scene, wherein each value in the object colour probability vector relates to a colour from a predefined set of colours and indicates a probability that a foreground object in the area of ​​the scene has the colour; and calculating, for each area of ​​the plurality of areas of the scene, a measure of variability of the probability indicated by the object colour probability vector associated with the area of ​​the scene and associating the measure of variability with the area of ​​the scene.

[0013] The method further includes providing a second video sequence showing the scene; tracking a foreground object in the second video sequence, where the tracked foreground object is in a first area of ​​a plurality of areas in the scene in a first image frame of the second video sequence and in a second different area of ​​the plurality of areas in the scene in a second image frame of the second video sequence; determining a first set of colors of the tracked foreground object in the first image frame and determining a second different set of colors of the tracked foreground object in the second image frame; and upon determining that the measure of variability associated with the first area of ​​the scene is lower than the measure of variability associated with the second area of ​​the scene, determining that the color(s) of the tracked foreground object are of the first set of colors, and otherwise determining that the color(s) of the tracked foreground object are of the second set of colors.

[0014] An "object color probability vector" is a mathematical representation associated with a particular area of ​​a scene. Each value in this vector corresponds to a color from a predefined set of colors. The predefined set of colors may include, for example, 8 colors, 12 colors, 16 colors, etc. The granularity of the predefined set of colors generally depends on the application. For example, too many shades of blue in the set may cause difficulty in specifying the actual blue color when searching for an object, for example, in a forensic search application, thus increasing the risk of false negatives in the search. On the other hand, too few colors may cause an increase in false positives, since all shades of blue that an object may have may fall into the color "blue". Each value in the object color probability vector indicates the likelihood or probability that a foreground object in that particular area of ​​the scene will possess the corresponding color, i.e., the estimated likelihood that a foreground object in the area of ​​the scene will be of the color corresponding to that value. It should be noted that each value in the object color probability vector may directly or indirectly represent a probability. A direct representation may, for example, include a percentage, e.g., a 23% probability that an object having the color "light yellow" will occur in an area of ​​a scene. An indirect representation may, for example, include a number of objects with a corresponding color detected in an area of ​​a scene, e.g., 15 objects having the color "dark red" detected in an area of ​​a scene. Such an indirect representation needs to be compared with other values ​​in the object color probability vector to determine the actual probability that an object having the color "dark red" occurs in an area of ​​a scene, e.g., 15 out of a total of 73 objects detected in an area of ​​a scene have the color "dark red", resulting in a probability of 20.5%.

[0015] The term "measure of variability" should be interpreted in the context of this specification as a statistical term referring to how spread out a set of data (object color probability vectors) is. The term indicates the degree of difference or dissimilarity in a data set, providing a sense of how much the numbers in the data set "vary" from the mean (or center point) and from each other. If all the numbers (i.e., probabilities) in the object color probability vector are close to each other and to the mean, the variability is low. If the numbers are spread over a wide range, the variability is high. For example, a variability measure for an object color probability vector containing values ​​[0.2, 0.3, 0.3, 0.2] is lower than a variability measure for an object color probability vector containing values ​​[0.1, 0.7, 0.1, 0.1]. In particular, in areas of a scene where objects exhibit a wide variety of colors, the variability measure is often lower. This is because the multiple colors contribute equally to the probability distribution, reflecting a uniform representation across the color spectrum. The measure is an aggregation based on multiple objects with different colors. As a result, when color variations are enormous and unbiased, the associated probabilities tend to converge, leading to lower variability since no single color significantly affects the overall measure.

[0016] As explained above, the apparent color of an object will be affected by ambient light sources, such as natural and artificial light sources. For example, if an area of ​​a scene is illuminated by yellow light, the apparent color of an object in that area of ​​the scene captured by an image may shift toward yellow, even if the natural color of the object is not yellow. In another example, an area of ​​a scene is shadowed by a building. The apparent color of an object in that area of ​​the scene captured by an image may shift toward darker colors (e.g., dark red, dark blue, black, etc.), even if the natural color of the object is not dark. For such areas in a scene where light sources and / or shadows may distort or affect the apparent color of an object toward a particular shade (color cast explained above), the variability measurement will be larger than for areas in a scene with more natural lighting. For example, if there is a yellow color cast due to yellow street lamps, a white car under the lamps will appear yellowish because the yellow light from the lamps is reflecting off the car. Similarly, a blue car will appear greener under yellow light because the blue of the car combined with the yellow light produces a greenish color. Thus, the probability that a foreground object in an area of ​​a scene (captured by an image) will be determined to have a yellow or green color may increase, and the probability that a foreground object in an area of ​​a scene will be determined to have a white or blue color may decrease, resulting in an increased measure of variability for areas of a scene under yellow street lamps compared to areas of a scene having only natural light.

[0017] The inventors have realised that such a measure of variability may be used to determine the colour(s) for a tracked object if a different set of colour(s) is determined for the object as it moves through the scene. The variability is determined during a construction (training) phase by detecting foreground objects in a video sequence (or several video sequences) capturing the scene, and using image data of the video sequence to determine colours for objects in different areas of the scene.

[0018] As explained above, a lower measure of variability associated with an area of ​​a scene indicates that the colors of objects in that area may be rendered more accurately in an image capturing the scene, compared to areas associated with a higher measure of variability. Thus, the measure of variability may be used as a color rendering metric associated with a particular area of ​​a scene. The term "color rendering metric" refers to a quantitative measure of the ability of a light source(s) (i.e., a light source that illuminates an area of ​​a scene, and possibly influenced by other objects that cast shadows, etc.) to accurately reveal the colors of various objects, compared to an ideal or natural light source. A lower measure of variability results in a higher color rendering metric, and vice versa.

[0019] Thus, the variability determined during the construction stage may be used for further video sequences capturing the scene, as described herein, to determine the "real" colors of objects moving through the scene. Advantageously, the most likely color (set of colors) for the object may be determined.

[0020] In some embodiments, all occurrences of the tracked foreground object in the second video sequence are labeled with the determined color(s). Advantageously, as explained above, images showing the foreground object when in an area of ​​the scene whose apparent color has been distorted away from its natural color may also be flagged when searching for objects of that particular color in the second video sequence.

[0021] In some embodiments, each of the first set of colors and the second set of colors includes a single color value, multiple color values, or one of multiple color values, each associated with a probability that the tracked object has the color. For example, the set of colors may be determined using a neural network trained to output multiple color values, each associated with a probability that the tracked object has the color, based on input pixel data indicative of the object. The multiple color values ​​are generally limited to a predefined set of colors. In another example, the pixel data indicative of the object may be used to calculate an average color, for example, all red, green, and blue values ​​from the RGB pixel data may be summed separately and then divided by the number of pixels to obtain an average red, green, and blue value. The resulting RGB value may then be used as a single color value. In another embodiment, color quantization or clustering may be used to group similar colors together to determine multiple color values ​​(the most common colors) for the object.

[0022] When the first video sequence and the second video sequence are captured by a camera with the same field of view in a scene, an area of ​​the scene corresponds to the same pixel region in the image frames of the first video sequence and the second video sequence. The granularity of the area of ​​the scene depends on the requirements and limitations of the application. For example, fewer areas of the scene may result in fewer computational resources being used to implement the method. On the other hand, increasing the number of areas in the scene may result in a more accurate final result (of the determined color(s) of the foreground object) since the likelihood of finding an area of ​​the scene with favorable lighting may increase. In some embodiments, each area of ​​the scene corresponds to a single pixel coordinate in the image frames of the first video sequence and the second video sequence.

[0023] If the first and second video sequences are captured by cameras with varying fields of view in the scene, the method further includes determining pixel regions in the image frames from the first or second video sequences that correspond to areas of the scene using camera parameters, the camera parameters including one or more of pan, tilt, roll, or zoom. Thus, the techniques described herein may be used for videos captured by a moving camera.

[0024] In some embodiments, detecting foreground objects in the image frames includes determining a location and extent of each detected foreground object in the image frames, the location and extent comprising one of a pixel mask or a bounding box. The location and extent may be used to map the object to an area of ​​the scene. If the bounding box / pixel mask of the object at least partially overlaps with two or more areas of the scene, the color of the object may contribute to an object color probability vector associated with each of these areas.

[0025] In some embodiments, the step of analyzing the first video sequence includes, for each foreground object in an area of ​​a scene in an image frame of the first video sequence, determining one or more colors of the foreground object from pixel data indicative of the foreground object in the image frame, and using the determined one or more colors when calculating an object color probability vector associated with the area of ​​the scene. For example, the pixel data indicative of the object may be used to calculate an average color, e.g., summing all red, green, and blue values ​​from the RGB pixel data separately (or similarly for other color spaces such as HSV, CIE, etc.) and then dividing by the number of pixels to obtain an average red, green, and blue value. The resulting RGB value may then be used as a single color value. In another embodiment, a color quantization or clustering technique that groups similar colors together may be used to determine multiple color values ​​(the most common colors) for the object. The one or more colors determined for the foreground object may all be part of a set of predefined colors. A color quantization technique may be used to map the color of the object to one of the predefined colors. The object color probability vector associated with the area of ​​the scene may then be updated using the determined color or colors of the foreground object, for example by updating the object count for each of the color or colors or by recalculating the probability based on the color or colors.

[0026] In an example, analyzing the first video sequence includes receiving, for each foreground object in an area of ​​a scene in an image frame of the first video sequence, a plurality of color values, each associated with a probability that the foreground object has a color in the image frame, and using the plurality of color values ​​and their associated probabilities when calculating an object color probability vector associated with the area of ​​the scene. The plurality of color values ​​may be received from a neural network trained to output a plurality of color values, each associated with a probability that the tracked object has a color, based on input pixel data indicative of the foreground object. The plurality of color values ​​are generally limited to a set of predefined colors. The object color probability vector associated with the area of ​​the scene may then be updated using the color values ​​and their respective associated probabilities.

[0027] In some examples, the step of calculating the object color probability vector for an area of ​​the plurality of areas in the scene includes detecting at least a threshold number of foreground objects in the area of ​​the scene. In some embodiments, at least a threshold number of each foreground object class of the plurality of foreground object classes needs to be detected. Thus, a sufficient number of objects in the area of ​​the scene can be analyzed to determine a representative object color probability vector for the area of ​​the scene. The threshold number depends on the requirements of the application as well as the scene captured by the video sequence. For example, the threshold number can be 50, 100, 130, 210, 450, etc.

[0028] In an embodiment, the first video sequence is captured during a first time period on one day, and the second video sequence is captured during a second time period on a subsequent day, with the second time period being completely encompassed within the first time period. Advantageously, the lighting conditions of the object color probability vector construction / training phase can be well adapted to the lighting conditions when the object color probability vector is used. The techniques described herein can be advantageously used for videos captured during daylight hours, because this increases the likelihood that at least some of the areas of the scene are illuminated so that the true color of the object is captured by the image. Thus, in some embodiments, the first video sequence (and the second video sequence) are captured during daylight hours.

[0029] In some examples, the measure of variability is at least one of the variance, standard deviation, mean absolute deviation, median absolute deviation, or coefficient of variation.

[0030] According to a second aspect of the present invention, the above object is achieved by a non-transitory computer-readable storage medium storing instructions for implementing the method according to the first aspect when executed on a device having processing capability.

[0031] According to a third aspect of the present invention, the above object is achieved by a system comprising one or more processors and one or more non-transitory computer-readable media storing first computer-executable instructions, which when executed by the one or more processors cause the system to perform the actions recited in the appended claims.

[0032] According to a fourth aspect of the present invention, the above object is achieved by a system comprising a colour matching system and a forensic search application, as detailed in the appended claims.

[0033] Advantageously, a forensic search application may then be used to search for the object, with the search request including a first color value. The first color value may be compared to the color(s) determined for the tracked foreground object, and if the first color value is determined to match the color determined for the foreground object using the techniques described herein, a search response may be returned based at least in part on the tracked foreground object. The details of the actual search response may be based on the requirements of the forensic search application. For example, the search response may include images in which an object having a color matching the color value of the search request was found, or a time span of the video stream in which the object was detected, or a timestamp in which the object was detected, the license plate of the object, the face of the object, etc.

[0034] The second, third and fourth aspects may generally have the same features and advantages as the first aspect. Furthermore, it should be noted that the present disclosure relates to all possible combinations of features unless otherwise specified. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 illustrates a scene in which light sources and other objects can cause uneven color casts in an image that captures the scene. [Diagram 2] FIG. 2 illustrates a system for calculation of a measure of variability for an area of ​​the scene in FIG. 1 according to an embodiment. [Diagram 3] FIG. 3 illustrates a system for using the variability measures from FIG. 2 to determine the color(s) of a tracked foreground object, according to an embodiment. [Figure 4] FIG. 4 illustrates a system comprising the color matching system according to FIGS. 2-3 and a forensic search application according to an embodiment. [Diagram 5] FIG. 2 illustrates a flowchart of a method for determining the color of a tracked object, according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] The color of an object in an image may be determined by the material and surface properties of the object, which determine the amount of light absorbed and reflected by the object, and therefore the perceived color of the object. The color of an object in an image may further be influenced by the camera settings (white balance, color profile, etc.) and image post-processing. These properties may generally be controllable by the owner of the camera. A property that may be difficult to control is the light source and lighting conditions when capturing an image or video, especially in a surveillance situation where the camera is continuously capturing a scene. The light source and lighting conditions may have a large effect on the detected (apparent, perceived, etc.) color of an object in an image of a scene, which means that the color may be perceived differently depending on the time of day, season, weather conditions, etc., as well as depending on where the object is in the scene when the image is captured. This may cause problems when searching for an object, for example, in a forensic search application based on the color of the object, or in other applications where it is important to know the color of the object.

[0037] The present disclosure aims to provide methods, systems, and software for determining the color of a tracked object in a video sequence of a scene when the lighting in the scene is non-uniform, which, as described herein, causes a non-uniform color cast in an image showing the scene, such that different parts of the image may have different hues.

[0038] 1 illustrates a scenario in which a scene 100 includes non-uniform lighting. The scene 100 includes a road for vehicles 108. Thus, the scene 100 includes objects 108 moving through the scene 100. The scene 100 further includes both natural lighting from the sun 102 and sky, and artificial lighting from street lamps 106. The scene also includes a tree 104 that casts a shadow over a portion of the scene 100.

[0039] The depicted scene 100 can therefore be divided into at least three areas 102a-102c, each distinguished by a unique lighting condition: the first area 102a is shaded by trees; the second area 102b is lit by street lamps 106; and finally, the third area 102c is exposed to both direct sunlight from the sun 102 and diffuse light from the sky.

[0040] It should be noted that the scene shown in FIG. 1 is simplified for ease of explanation, and that in reality a scene will typically contain many more objects and / or light sources that affect the lighting conditions in different areas of the scene.

[0041] A vehicle 108 moving on a road can be utilized to determine which area of ​​the scene 100 will yield the most accurate color of the objects (located in that area) when shown by an image / video stream capturing the scene 100. Next, the process of calculating color rendering metrics for an area of ​​a scene will be described in conjunction with Figures 2 and 5.

[0042] 5 shows a flow chart of a method 500 for determining the colour of a tracked object. The method includes training / construction stages S502 to S508 and implementation stages S510 to S516. The training stage will now be described in conjunction with FIG.

[0043] FIG. 2 shows a camera 202 capturing the scene from FIG. 1 and thus providing, at S502, a video sequence 204 showing the scene 100.

[0044] The image frames of the video sequence 204 each include a background object (background) and a foreground object 206, which in this simplified example is a vehicle 108 moving in the scene.

[0045] For each image frame of the multiple image frames of the video sequence 204, an object detector 212 is used to detect a foreground object 206 in the image frame at S504. Any suitable type of object detector, both neural network-based and non-neural network-based techniques, may be used. Neural network-based techniques include using convolutional neural networks using models such as YOLO (You Only Look Once), SSD (Single Shot Multibox Detector), DETR (End-to-End Detection with Transformers), and Faster R-CNN (Region-Based Convolutional Neural Network). Other neural network-based techniques include LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) networks. Non-neural network-based techniques include Histogram of Oriented Gradients (HOG), Scale-Invariant Feature Transform (SIFT), and Speeded Up Robust Features (SURF).

[0046] The detected foreground objects 206 are sent to a color probability calculator 214. The color probability calculator 214 may be configured to divide the scene into areas of the scene or to receive data indicative of the areas of the scene.

[0047] For a fixed camera 202 capturing the scene 100, each area of ​​the scene corresponds to the same pixel region in an image frame of the video sequence 204. For example, each area of ​​the scene may correspond to a 10*10 pixel region, or a 20*20 pixel region, or a 30*20 pixel region, or a pixel region of any suitable size of an image frame of the video sequence. In some embodiments, each area of ​​the scene corresponds to a single pixel coordinate in an image frame of the video sequence 204. The area of ​​the scene may be determined based on an analysis of the lighting conditions of the scene in some embodiments. In other embodiments, the area of ​​the scene may be determined based on a semantic segmentation of the background of the scene.

[0048] If the video sequence is captured by a camera with a varying field of view through the scene (not shown in FIG. 2), the camera parameters are used to determine pixel regions in image frames from the video sequence that correspond to areas of the scene, the camera parameters including one or more of pan, tilt, roll, or zoom.

[0049] In the simplified example of FIG. 2, the scene is captured by a fixed camera and is divided into three areas of the scene 102a, 102b, 102c according to the above description of FIG.

[0050] For each area of ​​the scene, the color probability calculator may calculate an object color probability vector 208a-208c associated with the area 102a-102c of the scene in S506. Each value in the object color probability vector 208a-208c relates to a color from a predefined set of colors and indicates the probability that a foreground object 206 in the area 102a-102c of the scene has the color.

[0051] The predefined colors may be configured based on the requirements of the system described herein. For example, the predefined colors may include 10-25 different colors, or 5-10 colors. An increase in the number of colors may require a long training phase of the system and techniques described herein. This is because a threshold number of foreground objects detected in a particular area of ​​a scene for calculating a representative object color probability vector 208a-208c may be correlated with the number of predefined colors. On the other hand, the accuracy of determining the most suitable area of ​​a scene for color determination (as described further below) may increase if the number of predefined colors increases. Thus, in some embodiments, calculating the object color probability vector 208a-208c for an area of ​​the plurality of areas 102a-102c in the scene 100 includes detecting at least a threshold number of foreground objects 206 in the area of ​​the scene. The threshold number may depend on the number of predefined colors, the number of moving objects 108 in the scene 100 in general per time interval, the variability of colors of such objects 108, etc.

[0052] Determining the color(s) of the foreground object 206 in the video sequence may be implemented with any suitable color determination algorithm. In some embodiments, analyzing the first video sequence includes, for each foreground object in an area of ​​a scene in an image frame of the first video sequence, determining one or more colors of the foreground object from pixel data of the foreground object in the image frame, and using the determined one or more colors when calculating an object color probability vector associated with the area of ​​the scene. The pixel data to use for a particular foreground object 206 may be determined by the object detector 212, which may determine the location and extent of each detected foreground object 206 in each image frame, the location and extent including one of a pixel mask, or a bounding box. Such pixel data may then be analyzed to determine one or more colors of the pixel data. For example, a histogram of colors may be determined and mapped to predetermined colors. The predetermined color having the most pixels in the pixel data with the same or similar color may be selected as the color of the object. In some cases, more than one color, e.g., the most common color in the pixel data, is determined for the object. Mapping the color of the pixel data to the predetermined colors may involve comparing the color of the pixel data to each of the colors among the predetermined colors to choose the "closest" color. The definition of closest may vary based on the color space, but generally it may be sufficient to calculate the Euclidean distance between the color to be mapped and each of the predetermined colors. If enough pixels (e.g., above a percentage threshold) are sufficiently similar (e.g., distance below a distance threshold), the object may be determined to have an associated predetermined color. The color determined for an object in an area of ​​the scene may get a "tick" (an increase in value / count) in the object color probability vector (associated with the area of ​​the scene in which the object is located) for that color.After the training phase, an object color probability vector for one area of ​​the scene may be something like (5 pre-determined colors, black, white, blue, red, green): [46, 120, 66, 40, 23], which may be converted to probabilities: [0.16, 0.41, 0.22, 0.13, 0.08]. For another area of ​​the scene, the numbers may be different based on the illumination conditions described above.

[0053] In other embodiments, a neural network may be used to determine the color of an object. For example, the neural network may be trained to determine the probability that an object has a certain color. Training may involve using training data, each training data including a set of pixels labeled with one or more of the pre-determined colors. The neural network may use this training data to learn how to identify the correct color (among the pre-determined colors) for a particular set of pixels. The output from using the neural network may be a number of color values, each associated with a probability that the foreground object has the color in the image frame. For example, the output for a dark blue object may yield the following probabilities (for five pre-determined colors: black, white, blue, red, green): [0.30, 0.01, 0.6, 0.05, 0.04]. Such vectors may be added to or used to recalculate the values ​​of the object probability vectors 208a-208c. For example, the object probability vectors 208a-208c may be normalized to always have a total value of 1 (100%) of the sum of all individual values ​​in the vector. In some embodiments, a color between the pre-determined colors that has a probability in the output from the neural network that exceeds a threshold probability for an input object may get a "tick" (an increase in value / count) in the object color probability vector for that color, as described above.

[0054] In some embodiments, an object is in two or more areas of a scene at the same time, for example, when the size of the object is larger than the size of the area of ​​the scene. In such a case, the determined color(s) of an object in an image frame of a video sequence may affect two or more object probability vectors. For example, as described above, detecting a foreground object in an image frame includes determining a location and range of each detected foreground object in the image frame, the location and range including one of a pixel mask or a bounding box. In such an example, all areas of the scene at least partially overlapped by the pixel mask / bounding box of an object detected in an image frame may be affected by the determined color for that object.

[0055] In the example of Figure 2, the number of pre-determined colors is three. The object probability vector 208a for area 102a (shadowed by tree 104 in scene 100) is determined to be [0.1, 0.1, 0.8]. The object probability vector 208b for area 102b (lit by street lamp 106 in scene 100) is determined to be [0.2, 0.7, 0.1]. The object probability vector 208c for area 102c (exposed to direct sunlight from both sun 102 and diffuse light from the sky in scene 100) is determined to be [0.3, 0.3, 0.4].

[0056] The color probability calculator 214 is further configured to calculate, at S508, a measure 210a-c of variability of the probabilities indicated by the object color probability vectors associated with the area of ​​the scene. As explained above, the measure of variability may be used as a "color rendering metric", e.g., a quantitative measure of the ability of the light source(s) in the area of ​​the scene (i.e., the light source that illuminates the area of ​​the scene, and possibly influenced by other objects that cast shadows, etc.) to accurately reveal the colors of various objects, as compared to an ideal or natural light source.

[0057] In Fig. 2, the variability measures 210a-210c are calculated using the variance of the respective object probability vectors 208a-208c. In other embodiments, other measures may be used, such as standard deviation, mean absolute deviation, median absolute deviation or coefficient of variation. In either case, a lower measure of variability for an area of ​​a scene (less variability between the probabilities that an object has its respective predetermined color) indicates an area of ​​the scene with suitable illumination for determining the "true" color of an object, because the low variability indicates that image data capturing the scene in that area of ​​the scene has not been tinted in a way that distorts the color distribution of the objects captured in that area of ​​the scene.

[0058] 2, the variance 210a of the shaded area 102a is 0.109, the variance 210b of the street-lit area 102b is 0.089, and the variance 210c of the naturally lit area 102c of the scene is 0.002. Thus, the third area 102c of the scene may be the most suitable (among the three areas 102a-102c) for determining the "true" color of the object.

[0059] Figure 3 shows how the metrics calculated in Figure 2 are used for a second video sequence 302 showing the scene 100 from Figure 1. The use of the metrics is now explained together with the implementation steps S510 to S516 of the method 500 shown in Figure 5.

[0060] At S510, a second video sequence 302 showing the scene 100 is provided. In FIG. 3, the second video sequence 302 includes three image frames 304a-304c. The image frames 304a-304c show a foreground object 306 moving in the scene. The image frames 304a-304c are input to an object tracker 308 for tracking the detected foreground object. There are several techniques used to track objects in a video sequence, ranging from classical computer vision techniques to modern deep learning techniques, using, for example, optical flow, Kalman filtering, particle filters, correlation filters, or the Deep SORT (Simple Online and Real-time Tracking by Deep Association Metric) algorithm, which uses a CNN to extract features and a Kalman filter to predict motion.

[0061] A foreground object 306 may be tracked in the second video sequence 302 by the object tracker 308 at S512. The tracked foreground object 306 is in a first area 102a of a plurality of areas 102a-102c in a scene in a first image frame 304a of the second video sequence, in a second different area 102b of the plurality of areas in the scene in a second image frame 304b, and in a third different area 102c of the plurality of areas in the scene in a third image frame 304c.

[0062] The system of FIG. 3 further comprises a color determination component 310 that may be used to determine the color(s) of the tracked object 306. The variability measures 210a-c for each area 102a-c of the scene calculated in FIG. 2 may be input to the color determination component 310. The color determination component 310 may be configured to determine a first set of colors of the tracked foreground object 306 in the first image frames 304a-c (when in the first area 102a of the scene), determine a second different set of colors of the tracked foreground object 306 in the second image frame 304b (when in the second area 102b of the scene), and determine a third different set of colors of the tracked foreground object 306 in the third image frame 304c (when in the third area 102c of the scene), at S512. The different sets of colors may be determined as described above. Thus, the set of colors may include a single color value, multiple color values, or one of multiple color values, each associated with a probability that the tracked object has the color.

[0063] Using the variability measurements 210a-210c, the color determination component 310 may select between the sets of colors in S516. More specifically, the color determination component 310 may select the set of colors determined for an object when the object is in an area of ​​the scene with a comparable lower measurement of variability. Thus, the color determination component 310 may determine that the color(s) 310 of the tracked object 306 are the set of colors determined for the object 306 when the object 306 is in an area of ​​the scene with a comparable lower measurement of variability. In this case, the color(s) 310 of the tracked object 306 are determined to be the set of colors determined for the object 306 in the third image frame 304c, i.e., the third set of colors.

[0064] In some embodiments, the color determination component 310 may be configured to label all occurrences of the tracked foreground object 306 in the second video sequence 302 with the determined color(s) 310. This may be particularly advantageous in a system that includes a color matching system 404 (described above in conjunction with FIGS. 2-3) and a forensic search application 402. The color matching system 404 may include components 212, 214, 308, 310 described above and may be configured to implement a function for determining the color(s) of the tracked object in the second video sequence (described above in conjunction with FIG. 3) based on a measure of variability of an area in the scene determined using the detected object in the first video sequence (described above in conjunction with FIG. 2). The color matching system 404 may be further configured to receive a search request 408 including a first color value, determine that the first color value matches the determined color for the foreground object, and return a search response 406 based at least in part on the foreground object.

[0065] The forensic search application 402 may be configured to provide a search request 408 including a first color value to a color matching system 404, receive a search response 406 from the color matching system 404, and display data from the search response to a user.

[0066] The color matching system 404 may be implemented in a single device, such as a camera, in an example. In other examples, some or all of the different components (modules, units, etc.) 212, 214, 308, 310 may be implemented in a server or in the cloud. In general, a device (camera, server, etc.) implementing the components 212, 214, 308, 310 may comprise the components 212, 214, 308, 310 and, more specifically, circuits configured to implement their functions. The described features of the color matching system 404 and the forensic search application 402 may be advantageously implemented in one or more computer programs executable on a programmable system including at least one programmable processor coupled to receive data and instructions from and transmit data and instructions to a data storage system, at least one input device, such as a camera, and, in some cases, at least one output device, such as a display. Suitable processors for the execution of a program of instructions include, by way of example, both general and special purpose microprocessors, and the sole processor or one of multiple processors or cores of any kind of computer, which processors may be supplemented by, or incorporated in, ASICs (Application Specific Integrated Circuits).

[0067] The above embodiments should be understood as illustrative examples of the present invention. Further embodiments of the present invention are envisaged. For example, the training phase may relate to a particular time span of a day, a particular time of year, a particular weather situation, etc. Variability measures determined for different time periods / weathers, etc. may be stored and implemented, for example, in color determination component 310 of FIG. 3, when similar time periods / weather conditions occur. In some embodiments, the variability measures of areas of a scene are periodically reset or recalibrated.

[0068] It will be understood that any feature described with respect to any one embodiment may be used alone or in combination with the other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Moreover, equivalents and modifications not described above may be employed without departing from the scope of the invention as defined in the appended claims.

Claims

1. 1. A computer-implemented method for determining a color of a tracked object, comprising: providing a first video sequence showing a scene and comprising a plurality of image frames; for each image frame of the plurality of image frames, detecting a foreground object in the image frame; analyzing the first video sequence and calculating, for each of a plurality of areas in the scene, an object color probability vector associated with the area of ​​the scene, wherein each value in the object color probability vector relates to a color from a predefined set of colors and indicates a probability that a foreground object in the area of ​​the scene has that color; calculating, for each of the plurality of areas of the scene, a measure of variability of the probabilities indicated by the object color probability vector associated with that area of ​​the scene, and associating the measure of variability with the area of ​​the scene; providing a second video sequence showing the scene; tracking a foreground object in the second video sequence, the tracked foreground object being in a first area of ​​the plurality of areas in the scene in a first image frame of the second video sequence and in a second, different area of ​​the plurality of areas in the scene in a second image frame of the second video sequence; determining a first set of colors for the tracked foreground object in the first image frame and determining a second, different set of colors for the tracked foreground object in the second image frame; if determining that the measure of variability associated with the first area of ​​the scene is lower than the measure of variability associated with a second area of ​​the scene, determining that the color of the tracked foreground object is of the first set of colors, and otherwise determining that the color of the tracked foreground object is of a second set of colors; Including, the first video sequence and the second video sequence are captured by a camera at the same field of view of the scene, and areas of the scene correspond to the same pixel regions in the image frames of the first video sequence and the second video sequence; or The first and second video sequences are captured by a camera at varying fields of view in the scene, wherein the method includes: using camera parameters to determine pixel regions in image frames from the first video sequence or the second video sequence that correspond to areas of the scene, the camera parameters including one or more of pan, tilt, roll, or zoom. The computer-implemented method further comprising:

2. The method of claim 1 , further comprising labeling all occurrences of the tracked foreground object in the second video sequence with the determined color.

3. Each of the first set of colors and the second set of colors comprises: A single color value, Multiple color values, or a plurality of color values, each associated with a probability that the tracked object has the color; The method of claim 1 , comprising one of:

4. The method of claim 1 , wherein each area of ​​the scene corresponds to a single pixel coordinate in the image frames of the first and second video sequences.

5. Detecting foreground objects in the image frames includes: determining a location and extent of each detected foreground object in the image frame, the location and extent comprising: Pixel mask, or bounding box including one of the following: The method of claim 1.

6. The step of analyzing the first video sequence includes, for each foreground object in the area of ​​the scene in an image frame of the first video sequence: determining one or more colors of the foreground object from pixel data representing the foreground object in the image frame; using the determined one or more colors when calculating the object color probability vector associated with the area of ​​the scene. The method of claim 1 , comprising:

7. The step of analyzing the first video sequence includes, for each foreground object in the area of ​​the scene in an image frame of the first video sequence: receiving a plurality of color values, each associated with a probability that the foreground object has the color in the image frame; using the plurality of color values ​​and their associated probabilities when calculating the object color probability vector associated with the area of ​​the scene; The method of claim 1 , comprising:

8. 2. The method of claim 1, wherein calculating the object color probability vector for an area of ​​the plurality of areas in the scene comprises detecting at least a threshold number of foreground objects in the area of ​​the scene.

9. 2. The method of claim 1, wherein the first video sequence is captured during a first time period on one day and the second video sequence is captured during a second time period on a subsequent day, the second time period being entirely contained within the first time period.

10. The measure of variability is Variance, standard deviation, mean absolute deviation, median absolute deviation, or coefficient of variation 10. The method of claim 9, wherein the at least one of

11. A non-transitory computer-readable storage medium storing instructions for implementing the method of any one of claims 1 to 10 when executed on a device having processing capabilities.

12. one or more processors; one or more non-transitory computer-readable media storing first computer-executable instructions; wherein the first computer-executable instructions, when executed by the one or more processors, provide the system with: providing a first video sequence showing a scene and comprising a plurality of image frames; For each image frame of the plurality of image frames, detecting a foreground object in the image frame; analyzing the first video sequence and calculating, for each of a plurality of areas in the scene, an object color probability vector associated with the area of ​​the scene, wherein each value in the object color probability vector relates to a color from a predefined set of colors and indicates a probability that a foreground object in the area of ​​the scene has that color; calculating, for each of the plurality of areas of the scene, a measure of variability of the probabilities indicated by the object color probability vector associated with the area of ​​the scene, and associating the measure of variability with the area of ​​the scene; providing a second video sequence showing the scene; tracking a foreground object in the second video sequence, the tracked foreground object being in a first area of ​​the plurality of areas in the scene in a first image frame of the second video sequence and in a second, different area of ​​the plurality of areas in the scene in a second image frame of the second video sequence; determining a first set of colors of the tracked foreground object in the first image frame and determining a second, different set of colors of the tracked foreground object in the second image frame; if determining that the measure of variability associated with the first area of ​​the scene is lower than the measure of variability associated with the second area of ​​the scene, determining that the color of the tracked foreground object is of the first set of colors, and otherwise determining that the color of the tracked foreground object is of a second set of colors; and execute actions including the first video sequence and the second video sequence are captured by a camera at the same field of view of the scene, and areas of the scene correspond to the same pixel regions in the image frames of the first video sequence and the second video sequence; or the first and second video sequences are captured by a camera with varying fields of view through the scene, and pixel regions in image frames from the first or second video sequences that correspond to areas of the scene are determined using camera parameters, the camera parameters including one or more of pan, tilt, roll, or zoom; system.

13. 1. A system comprising a color matching system and a forensic search application, the color matching system comprising: one or more processors; and one or more non-transitory computer-readable media storing first computer-executable instructions, which when executed by the one or more processors, provide the system with: providing a first video sequence showing a scene and comprising a plurality of image frames; For each image frame of the plurality of image frames, detecting a foreground object in the image frame; analyzing the first video sequence and calculating, for each of a plurality of areas in the scene, an object color probability vector associated with the area of ​​the scene, wherein each value in the object color probability vector relates to a color from a predefined set of colors and indicates a probability that a foreground object in the area of ​​the scene has that color; calculating, for each of the plurality of areas of the scene, a measure of variability of the probabilities indicated by the object color probability vector associated with the area of ​​the scene, and associating the measure of variability with the area of ​​the scene; providing a second video sequence showing the scene; tracking a foreground object in the second video sequence, the tracked foreground object being in a first area of ​​the plurality of areas in the scene in a first image frame of the second video sequence and in a second, different area of ​​the plurality of areas in the scene in a second image frame of the second video sequence; determining a first set of colors of the tracked foreground object in the first image frame and determining a second, different set of colors of the tracked foreground object in the second image frame; determining that the color of the tracked foreground object is of the first set of colors if the measure of variability associated with the first area of ​​the scene is lower than the measure of variability associated with the second area of ​​the scene, and otherwise determining that the color of the tracked foreground object is of the second set of colors; receiving a search request including a first color value; determining that the first color value matches a color determined for the foreground object; returning a search response based at least in part on the foreground object; and execute actions including the forensic search application: one or more processors; and one or more non-transitory computer-readable media storing second computer-executable instructions, the second computer-executable instructions, when executed by the one or more processors, causing the forensic search application to: providing the search request including the first color value to the color matching system; receiving a search response from the color matching system; displaying data from the search response to a user; and and execute actions including the first video sequence and the second video sequence are captured by a camera at the same field of view of the scene, and areas of the scene correspond to the same pixel regions in the image frames of the first video sequence and the second video sequence; or the first and second video sequences are captured by a camera with varying fields of view through the scene, and pixel regions in image frames from the first or second video sequences that correspond to areas of the scene are determined using camera parameters, the camera parameters including one or more of pan, tilt, roll, or zoom; system.

Images