Method for determining color of tracked object

Abstract

The present disclosure relates to a method (500) for determining a color of a tracked object. Using the first video sequence and foreground objects detected therein, a color rendering metric may be determined for each of a plurality of regions in the scene. Then, in the event that the tracked object is determined to have different colors in different images of the second video sequence, the color rendering metric may be used such that a color detected in a region associated with a higher color rendering metric is selected in a region associated with a lower color rendering metric.

Description

Technical Field

[0001] The present invention relates to determining the color of a tracked object in a video sequence depicting a scene, and more particularly to determining the color of a tracked object in a video sequence depicting a scene with varying lighting conditions. Background Art

[0002] A color cast refers to the overall hue of a specific color throughout an image, usually as a result of specific lighting conditions. Uneven color cast is a variation of this phenomenon, referring to different hues that appear in different areas of an image. This occurs when different light sources with different color temperatures illuminate different aspects of a scene. For example, in an outdoor environment, daylight and street lighting may coexist, resulting in mixed lighting conditions. Daylight may cast a cooler, bluer light in some parts of the scene, while street lighting may apply a warmer, yellower light in other parts. In addition to this, shadows can also introduce localized color casts. The color of the light in the shadows is usually different from the direct light, resulting in an uneven color cast.

[0003] An example of an application is that in surveillance systems, it is very important to deal with uneven color casts, especially when an operator examines a video sequence to locate an object of a specific color. Manually screening multiple video sequences to identify items of interest is tedious and time consuming. To address this problem, automatic video analysis tools are often used to analyze video sequences. These tools can detect objects in a sequence and annotate the detected objects with properties such as object type, size, and speed. These features can then be exploited by the operator when searching for a specific object of interest (for example, in forensic search applications). When searching for an object in an image or video sequence, an important feature is the color of the object. However, the apparent color of the object will be affected by surrounding light sources such as natural and artificial light sources, which may lead to a false impression of the color of the object in the image. An example of this situation is a white car under a yellow street light, which may cause the color of the car in the image capturing the scene to be yellow instead of white.

[0004] Another problem is that lighting conditions in a scene vary based on many different reasons, such as the location of active artificial light sources, time of day, time of year, weather conditions, buildings, vegetation, etc. This makes it difficult to accurately determine the actual or natural color of an object captured in an image, which in turn can negatively impact the possibility of searching for an object based on its color.

[0005] US 2007 / 154088 discusses color recognition in digital images, with the goal of determining the robust perceived color or true color of an object, which is defined as the color of the object perceived under certain standard viewing, lighting and sensing conditions.

[0006] CN107292933 describes the use of a neural network to determine color.

[0007] Thierry Bouwmans et al., “On the role and the importance of features for background modelling and foreground detection” (Arxiv.org, Cornell University Library, November 28, 2016, XP080735023), describes modeling and color changes for detection purposes.

[0008] Therefore, improvements are needed in this regard. Summary of the invention

[0009] In view of the above, it would be beneficial to solve or at least reduce one or several of the above mentioned disadvantages, as set out in the attached independent patent claims.

[0010] According to a first aspect of the invention, a computer-implemented method for determining the color of a tracked object is provided, comprising the following steps: providing a first video sequence depicting a scene, the first video sequence comprising a plurality of image frames; for each of the plurality of image frames, detecting a foreground object in the image frame; for each of a plurality of regions in the scene, analyzing the first video sequence and calculating an object color probability vector associated with the region of the scene, wherein each value in the object color probability vector is associated with a color in a set of predefined colors and indicates a probability that a foreground object located in the region of the scene has the color; for each of the plurality of regions of the scene, calculating a variability measure of the probability indicated by the object color probability vector associated with the region of the scene, and associating the variability measure with the region of the scene.

[0011] The method further includes: providing a second video sequence depicting a scene; tracking a foreground object in the second video sequence, the tracked foreground object being located in a first area of ​​a plurality of areas in the scene in a first image frame of the second video sequence and being located in a different second area of ​​a plurality of areas in the scene in a second image frame of the second video sequence; determining a first set of colors for the tracked foreground object in the first image frame and determining a different second set of colors for the tracked foreground object in the second image frame; and determining that the colors of the tracked foreground object are the first set of colors when it is determined that a variability measurement result associated with the first area of ​​the scene is lower than a variability measurement result associated with the second area of ​​the scene, otherwise determining that the colors of the tracked foreground object are the second set of colors.

[0012] An "object color probability vector" is a mathematical representation associated with a particular region of a scene. Each value within the vector corresponds to a color in a set of predefined colors. The set of predefined colors may include, for example, 8 colors, 12 colors, 16 colors, etc. The granularity of the set of predefined colors generally depends on the application. For example, too many shades of blue in the set may make it difficult to specify the actual blue when searching for an object, for example in a forensic search application, and therefore, increase the risk of false negatives in the search. On the other hand, too few colors may lead to an increase in false positives, because all shades of blue that an object may have may belong to "blue". Each value in the object color probability vector indicates the likelihood or probability that a foreground object located within a particular region of the scene has the corresponding color, that is, indicates the estimated likelihood that a foreground object in that region of the scene will have a color corresponding to that value. It should be noted that each value in the object color probability vector may represent a probability directly or indirectly. For example, a direct representation may include the percentage of objects with a "light yellow" color that will appear in a region of the scene, for example, a probability of 23%. For example, an indirect representation may include a count of objects with a corresponding color detected in a region of the scene, for example, 15 objects with a "dark red" color were detected in a region of the scene. This indirect representation needs to be compared with other values ​​in the object color probability vector to determine the actual probability of an object with the color "dark red" appearing in the region of the scene, e.g., 15 of a total of 73 objects detected in the region of the scene have the color "dark red", resulting in a probability of 20.5%.

[0013] In the context of this specification, the term "variability measurement result" should be interpreted as a statistical term that refers to how a set of data (object color probability vector) is spread out. It indicates the degree of difference or dissimilarity in the data set, providing the degree of "difference" between the numbers in the data set and the average (or center point) and each other. If all the numbers (i.e., probabilities) in the object color probability vector are close to each other and close to the average, the variability is low. If the numbers are widely distributed, the variability is high. For example, the variability measurement result of the object color probability vector including the values ​​[0.2, 0.3, 0.3, 0.2] is lower than the variability measurement result of the object color probability vector including the values ​​[0.1, 0.7, 0.1, 0.1]. Specifically, in areas of the scene where objects present various colors, the variability measurement result is generally low. This is because multiple colors contribute equally to the probability distribution, reflecting a uniform distribution in the color spectrum. The measurement result is a collection of multiple objects based on different colors. Therefore, when the color changes are large and balanced, the associated probabilities tend to converge, resulting in lower variability because no single color will have a significant impact on the overall measurement.

[0014] As described above, the apparent color of an object will be affected by surrounding light sources such as natural and artificial light sources. For example, if an area of ​​a scene is illuminated by yellow light, the apparent color of an object in that area of ​​the scene captured by the image may be shifted toward yellow, even if the natural color of the object is not yellow. In another example, an area of ​​the scene is blocked by a building. The apparent color of an object in that area of ​​the scene captured by the image may be shifted toward a darker color (e.g., dark red, dark blue, black, etc.), even if the natural color of the object is not dark. For these areas in the scene, the light source and / or shadow may tilt the apparent color of the object toward a specific hue or affect it (color shift as described above), and the variability measurement result will be greater than the area with more natural light in the scene. For example, if there is a yellow shift due to a yellow street light, a white car under the light will look yellow because the yellow light from the light will be reflected by the car. Similarly, a blue car looks greener under yellow light because the blue color of the car combined with the yellow light will produce a green color. Thus, the probability of foreground objects located in areas of the scene (captured by the image) being determined to have a yellow or green color may increase, while the probability of foreground objects located in areas of the scene being determined to have a white or blue color may decrease, resulting in increased variability measurements for areas of the scene under yellow street lights compared to areas of the scene with only natural light.

[0015] The inventors have realized that such variability measurements can be used to determine the color of a tracked object, where different color groups (one or more colors) are determined for an object as it moves through a scene. Variability is determined during a configuration (training) phase by detecting foreground objects in a video sequence (or several video sequences) capturing a scene and using the image data of the video sequences to determine the color of objects located in different areas of the scene.

[0016] As described above, a lower variability measurement associated with an area of ​​a scene indicates that the colors of objects located in that area can be more accurately rendered in an image of the captured scene than an area associated with a higher variability measurement. Thus, the variability measurement can be used as a color rendering metric associated with a particular area of ​​a scene. The term "color rendering metric" refers to a quantitative measure of the ability of a light source (i.e., a light source illuminating an area of ​​a scene may be affected by other objects to produce shadows, etc.) to accurately display the colors of various objects compared to an ideal or natural light source. A lower variability measurement results in a higher color rendering metric, and vice versa.

[0017] Thus, the variability determined during the configuration phase can be used in another video sequence captured of the scene to determine the "actual" color of an object moving through the scene, as described herein. Advantageously, the most likely color (color set) of an object can be determined.

[0018] In some embodiments, all occurrences of the tracked foreground object in the second video sequence are marked with the determined color. Advantageously, when searching for an object of a particular color in the second video sequence, as described above, images showing foreground objects may also be marked when in an area of ​​the scene where the apparent color deviates from its natural color.

[0019] In some embodiments, each of the first and second sets of colors includes one of the following: a single color value; a plurality of color values; and a plurality of color values, each color value being associated with the probability that the tracked object has the color. For example, a neural network can be used to determine the set of colors, and the neural network is trained to output a plurality of color values ​​based on the input pixel data depicting the object, each color value being associated with the probability that the tracked object has the color. Multiple color values ​​are generally limited to a set of predefined colors. In another example, the pixel data depicting the object can be used to calculate the average color, for example, all red, green, and blue values ​​from the RGB pixel data are added together, and then divided by the number of pixels to obtain the average value of red, green, and blue. The resulting RGB value can then be used as a single color value. In another embodiment, color quantization or clustering that groups similar colors together can be used to determine multiple color values ​​(the most common colors) of an object.

[0020] In the case where the first video sequence and the second video sequence are captured by cameras having the same field of view in the scene, the area of ​​the scene corresponds to the same pixel area in the image frames of the first video sequence and the second video sequence. The granularity of the area of ​​the scene depends on the requirements of the application and the limitations of the application. For example, fewer areas of the scene can result in fewer computing resources for implementing the method. On the other hand, increasing the number of areas in the scene can result in a more accurate final result (the determined color of the foreground object) because the possibility of finding an area with favorable lighting of the scene can be increased. In some embodiments, each area of ​​the scene corresponds to a single pixel coordinate in the image frames of the first video sequence and the second video sequence.

[0021] In the case where the first video sequence and the second video sequence are captured by a camera with a changing field of view in the scene, the method further comprises the steps of determining a pixel area corresponding to an area of ​​the scene in an image frame from the first video sequence or the second video sequence using camera parameters, the camera parameters comprising one or more of pan, tilt, roll, and zoom. Thus, the techniques described herein may also be used for videos captured by a moving camera.

[0022] In some embodiments, the step of detecting foreground objects in the image frame includes: determining the position and range of each detected foreground object in the image frame, wherein the position and the range include one of a pixel mask and a bounding box. The position and the range can be used to map the object to a specific area of ​​the scene. If the bounding box / pixel mask of the object at least partially overlaps with more than one area of ​​the scene, the color of the object can contribute to the object color probability vector associated with each of these areas.

[0023] In some embodiments, the step of analyzing the first video sequence includes, for each foreground object in the region of the scene in the image frame of the first video sequence: determining one or more colors of the foreground object from the pixel data depicting the foreground object in the image frame; and using the determined one or more colors when calculating the object color probability vector associated with the region of the scene. For example, the pixel data describing the object can be used to calculate the average color, for example, all red, green and blue values ​​from the RGB pixel data are added separately (or similarly for other color spaces, such as HSV, CIE), and then divided by the number of pixels to obtain the average value of red, green and blue. The resulting RGB value can then be used as a single color value. In another embodiment, color quantization or clustering that groups similar colors together can be used to determine multiple color values ​​(the most common colors) of the object. The one or more colors determined for the foreground object can all be part of the set of predefined colors. Color quantization techniques can be used to map the color of the object to one of the predefined colors. The determined one or more colors of the foreground object can then be used to update the object color probability vector associated with the region of the scene, for example, by updating the object count for each of the one or more colors, or by recalculating the probability based on the one or more colors.

[0024] In an example, the step of analyzing a first video sequence includes, for each foreground object located in a region of a scene in an image frame of the first video sequence: receiving a plurality of color values, each color value being associated with a probability that the foreground object in the image frame has that color; and using the plurality of color values ​​and their associated probabilities when calculating an object color probability vector associated with the region of the scene. The plurality of color values ​​may be received from a neural network that is trained to output a plurality of color values ​​based on input pixel data depicting the foreground object, each color value being associated with a probability that the tracked object has that color. The plurality of color values ​​are typically limited to a set of predefined colors. The color values ​​and their respective associated probabilities may then be used to update the object color probability vector associated with the region of the scene.

[0025] In some examples, the step of calculating the object color probability vector of a region in a plurality of regions in a scene includes: detecting at least a threshold number of foreground objects in the region of the scene. In some embodiments, it is necessary to detect at least a threshold number of each foreground object class in a plurality of foreground object classes. Thus, a sufficient number of objects located in the region of the scene can be analyzed to determine a representative object color probability vector for the region of the scene. The threshold number depends on the requirements of the application and the scene captured by the video sequence. For example, the threshold number can be 50, 100, 130, 210, 450, etc.

[0026] In an embodiment, the first video sequence is captured during a first time period of a day, wherein the second video sequence is captured during a second time period of a subsequent day, wherein the second time period is completely contained within the first time period. Advantageously, when using object color probability vectors, the lighting conditions of the configuration / training phase of the object color probability vectors can be sufficiently good to match the lighting conditions. The techniques described herein can be advantageously used for videos captured during the day, because this increases the likelihood that at least some of the areas of the scene are illuminated so that the true color of the object is captured by the image. Therefore, in some embodiments, the first video sequence (and the second video sequence) are captured during the day.

[0027] In some examples, the variability measure is at least one of variance, standard deviation, mean absolute deviation, mean absolute deviation, and coefficient of variation.

[0028] According to a second aspect of the present invention, the above object is achieved by a non-transitory computer-readable storage medium having instructions stored thereon, which are used to implement the method according to the first aspect when executed on a device having processing capabilities.

[0029] According to the third aspect of the present invention, the above-mentioned purpose is achieved by a system, which includes: one or more processors; and one or more non-temporary computer-readable media storing first computer-executable instructions, which, when executed by one or more processors, enable the system to perform actions as described in detail in the attached claims.

[0030] According to a fourth aspect of the present invention, the above objects are achieved by a system comprising a color matching system and a forensic search application, as described in detail in the accompanying claims.

[0031] Advantageously, the forensic search application can then be used to search for the object, wherein the search request includes the first color value. The first color value can be compared to a color determined for a tracked foreground object, and if it is determined that the first color value matches the color determined for the foreground object using the techniques described herein, a search response can be returned based at least in part on the tracked foreground object. The details of the actual search response can be based on the requirements of the forensic search application. For example, the search response can include finding an image of an object having a color that matches the color value of the search request, or a time span of a video stream in which the object was detected, or a timestamp of when the object was detected, a license plate of the object, a face of the object, etc.

[0032] The second, third and fourth aspects may generally have the same features and advantages as the first aspect. It should also be noted that the present disclosure relates to all possible combinations of features, unless explicitly stated otherwise. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A scene is shown in which light sources and other objects may cause uneven color casts in an image captured of the scene;

[0034] Figure 2 A method for calculating the Figure 1 A system for measuring the variability of the regions of a scene;

[0035] Figure 3 The use of the embodiment from Figure 2 A system for determining the color of a tracked foreground object by measuring the variability of the foreground object;

[0036] Figure 4 According to an embodiment, a method including Figures 2 to 3 Color matching systems and systems for forensic search applications;

[0037] Figure 5 A flow chart illustrating a method for determining the color of a tracked object according to an embodiment. DETAILED DESCRIPTION

[0038] The color of an object in an image can be determined by the material and surface properties of the object, which determine the amount of light absorbed and reflected by the object, and therefore, its perceived color. The color of an object in an image may be further affected by camera settings (white balance, color profile, etc.) and image post-processing. These properties can usually be controlled by the owner of the camera. Attributes that may be difficult to control are the light source and lighting conditions when capturing images or videos, especially in surveillance situations where the camera continuously captures scenes. The light source and lighting conditions may have a significant impact on the detected (apparent, perceived, etc.) color of an object in an image of a scene. This in turn means that the color may be perceived differently depending on the time of day, the time of the year, weather conditions, etc. and depending on the position of the object in the scene when the image was captured. This may cause problems when searching for objects, such as in forensic search applications based on object color, or in other applications where knowing the color of an object is important.

[0039] The present disclosure is directed to methods, systems, and software for determining the color of a tracked object in a video sequence of a scene when the illumination in the scene is uneven. As described herein, uneven illumination may result in uneven color casts in an image depicting a scene, resulting in different parts of the image having different hues.

[0040] Figure 1 The scene 100 is shown to include a situation where uneven lighting is present. The scene 100 includes a road for a vehicle 108. Thus, the scene 100 includes an object 108 moving in the scene 100. The scene 100 further includes two natural lightings from the sun 102 and the sky and artificial lighting from a street lamp 106. The scene also includes a tree 104, which casts a shadow on a portion of the scene 100. Thus, the depicted scene 100 can be divided into at least three regions 102a to 102c, each region being distinguished by a unique lighting condition. The first region 102a is shaded by a tree. The second region 102b is illuminated by a street lamp 106. Finally, the third region 102c is exposed to direct sunlight from the sun 102 and diffuse light from the sky.

[0041] It should be noted that for ease of description, Figure 1 , and in reality, a scene typically includes many more objects and / or light sources that affect the lighting conditions of different areas of the scene.

[0042] When depicted by an image / video stream capturing scene 100, a vehicle 108 moving on a road can be used to determine which area of ​​scene 100 produces the most accurate color of an object (located in that area). Figure 2 and Figure 5 Explain how the color rendering metrics for a region of the scene are calculated.

[0043] Figure 5 A flow chart of a method 500 for determining the color of a tracked object is shown. The method includes a training / configuration phase S502 to S508 and an implementation phase S510 to S516. Figure 2 Describe the training phase.

[0044] Figure 2 The camera 202 is shown capturing Figure 1 , and therefore, providing S502 a video sequence 204 depicting the scene 100 .

[0045] The image frames of the video sequence 204 each include a background object (background) and a foreground object 206. In this simplified example, the foreground object 206 is a vehicle 108 moving in the scene.

[0046] The object detector 212 is used to detect S504 a foreground object 206 in an image frame for each of a plurality of image frames of the video sequence 204. Any suitable type of object detector (both neural network-based techniques and non-neural network-based techniques) may be used. Neural network-based techniques include using convolutional neural networks, using models such as YOLO (YouOnly Look Once), SSD (Single Shot Multi-Box Detector), DETR (End-to-End Detection Using Transformers), and Faster R-CNN (Region-based Convolutional Neural Network). Other neural network-based techniques include LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) networks. Non-neural network-based techniques include histograms of oriented gradients (HOG), scale-invariant feature transforms (SIFT), and speeded-up robust features (SURF).

[0047] The detected foreground objects 206 are sent to a color probability calculator 214. The color probability calculator 214 may be configured to divide the scene into regions of the scene or to receive data indicative of regions of the scene.

[0048] In the case where the fixed camera 202 captures the scene 100, each region of the scene corresponds to the same pixel region in the image frame of the video sequence 204. For example, each region of the scene may correspond to a 10×10 pixel region, or a 20×20 pixel region, or a 30×20 pixel region, or a pixel region of any suitable size in the image frame of the video sequence. In some embodiments, each region of the scene corresponds to a single pixel coordinate in the image frame of the video sequence 204. In some embodiments, the region of the scene may be determined based on an analysis of the lighting conditions of the scene. In other embodiments, the region of the scene may be determined based on a semantic segmentation of the background of the scene.

[0049] In the case where the video sequence is captured by a camera with a changing field of view in the scene ( Figure 2 (not shown) , a pixel area corresponding to an area of ​​the scene in an image frame from a video sequence is determined using camera parameters, the camera parameters including one or more of translation, tilt, roll, and zoom.

[0050] According to the above Figure 1 Description, in Figure 2 In the simplified example of , a scene is captured by a fixed camera and is divided into three regions 102a, 102b, 102c of the scene.

[0051] For each region of the scene, the color probability calculator may calculate S506 an object color probability vector 208a to 208c associated with the region 102a to 102c of the scene. Each value in the object color probability vector 208a to 208c is associated with a color from a set of predefined colors and indicates the probability that a foreground object 206 located in the region 102a to 102c of the scene has that color.

[0052] Predefined colors can be configured based on the requirements of the system described in this article. For example, predefined colors can include 10 to 25 different colors or 5 to 10 colors. The increase in the number of colors may require an extended training phase of the system and technology described in this article. This is because in order to calculate representative object color probability vectors 208a to 208c, the threshold number of foreground objects detected in a specific area of ​​a scene can be related to the number of predefined colors. On the other hand, if the number of predefined colors increases, the accuracy of the most suitable area determined for color of the scene (as will be further described below) can be increased. Therefore, in some embodiments, the object color probability vectors 208a to 208c of the area in the multiple areas 102a to 102c in the calculation scene 100 include at least the threshold number of foreground objects 206 in the area of ​​the detection scene. The threshold number can depend on the number of predefined colors, the number of moving objects 108 usually located in the scene 100 per time interval, the variability of the colors of these objects 108, etc.

[0053] The determination of the color of the foreground object 206 in the video sequence can be implemented using any suitable color determination algorithm. In some embodiments, the step of analyzing the first video sequence includes, for each foreground object in the region of the scene in the image frame of the first video sequence: determining one or more colors of the foreground object from the pixel data of the foreground object in the image frame; and using the determined one or more colors when calculating the object color probability vector associated with the region of the scene. The pixel data for a particular foreground object 206 can be determined by an object detector 212, which can determine the position and range of each detected foreground object 206 in each image frame, wherein the position and range include one of a pixel mask and a bounding box. Therefore, such pixel data can be analyzed to determine one or more colors of the pixel data. For example, a color histogram can be determined and mapped to a predefined color. The predefined color of the majority of pixels in the pixel data having the same color or a similar color can be selected as the color of the object. In some cases, more than one color is determined for the object, for example, the most common color in the pixel data. Mapping the color of the pixel data to the predefined color can include comparing the color of the pixel data with each color among the predefined colors to select the "closest" one color. The definition of closest can vary based on the color space, but in general, it may be sufficient to calculate the Euclidean distance between the color to be mapped and each of the predefined colors. If enough pixels (e.g., above a certain percentage threshold) are similar enough (e.g., the distance is below a distance threshold), it can be determined that the object has an associated predefined color. The color determined for an object located in an area of ​​the scene can get a "tick" (increase in value / count) in the object color probability vector for that color (associated with the area of ​​the scene where the object is located). After the training phase, the object color probability vector for a certain area of ​​the scene may look like this (5 predefined colors, black, white, blue, red, green): [46, 120, 66, 40, 23], which can be converted to probabilities: [0.16, 0.41, 0.22, 0.13, 0.08]. For another area of ​​the scene, these numbers may be different based on the lighting conditions as described above.

[0054] In other embodiments, a neural network may be used to determine the color of an object. For example, a neural network may be trained to determine the probability that an object has a certain color. Training may include using training data, each of which includes a group of pixels that are labeled with one or more predefined colors. The neural network may use the training data to learn how to identify the correct color (among the predefined colors) of a particular group of pixels. The output of using the neural network may be a plurality of color values, each of which is associated with the probability that a foreground object in an image frame has that color. For example, the output for a dark blue object may result in the following probabilities (for 5 predefined colors, black, white, blue, red, green): [0.30, 0.01, 0.6, 0.05, 0.04]. This vector may be used to join the values ​​of the object probability vectors 208a to 208c, or to recalculate the values ​​of the object probability vectors 208a to 208c. For example, the object probability vectors 208a to 208c may be normalized to always have a total value of one (100%) that is the sum of all individual values ​​in the vector. In some embodiments, as described above, colors among the predefined colors that have a probability in the output from the neural network that is above a threshold probability for a certain input object may get a "tick" (increase in value / count) in the object color probability vector for that color / these colors.

[0055] In some embodiments, an object is located in more than one area of ​​a scene simultaneously, for example, when the size of the object is larger than the size of an area of ​​the scene. In this case, the determined color of the object in the image frame of the video sequence may affect more than one object probability vector. For example, as described above, the step of detecting foreground objects in the image frame includes determining the position and range of each detected foreground object in the image frame, wherein the position and the range include one of a pixel mask and a bounding box. In such an example, all areas of the scene that are at least partially overlapped by the pixel mask / bounding box of an object detected in the image frame may be affected by the color determined for the object.

[0056] exist Figure 2 In the example of , the number of predefined colors is three (3). The object probability vector 208a of the area 102a (obscured by the tree 104 in the scene 100) is determined to be [0.1, 0.1, 0.8]. The object probability vector 208b of the area 102b (illuminated by the street lamp 108 in the scene 100) is determined to be [0.2, 0.7, 0.1]. The object probability vector 208c of the area 102c (exposed to direct sunlight from the sun 102 and diffuse light from the sky in the scene 100) is determined to be [0.3, 0.3, 0.4].

[0057] The color probability calculator 214 is further configured to calculate S508 variability measures 210a to 210c of the probabilities indicated by the object color probability vectors associated with the region of the scene. As described above, the variability measures can be used as a color rendering metric, for example, a quantitative measure of the ability of a light source in a region of the scene (i.e., a light source that illuminates a region of the scene, which may be affected by other objects that cause shadows, etc.) to accurately display the colors of various objects compared to an ideal or natural light source.

[0058] exist Figure 2 , variability measurements 210a to 210c are calculated using the variance of the corresponding object probability vectors 208a to 208c. In other embodiments, other measurements may be used, such as standard deviation, mean absolute deviation, median absolute deviation, or coefficient of variation. In any event, a lower variability measurement for a region of a scene (less variability in the probabilities of objects having corresponding predefined colors) indicates a region of the scene with suitable lighting for determining the "true" color of an object, because the lower variability indicates that the image data captured at that region of the scene was not colored in a manner that affects the accuracy of the color distribution of objects captured at that region of the scene.

[0059] exist Figure 2 , the variance 210a of the shadowed area 102a is 0.109, the variance 210b of the streetlight illuminated area 102b is 0.089, and the variance 210c of the area 102c of the scene illuminated by natural light is 0.002. Therefore, the third area 102c of the scene may be the most suitable area (out of the three areas 102a-102c) for determining the "true" color of the object.

[0060] Figure 3 Show Figure 2 How the metrics calculated in are used to depict Figure 1 The second video sequence 302 of the scene 100 in FIG. Figure 5 The implementation stages S510 to S516 of the method 500 are shown to describe the use of the metric.

[0061] A second video sequence 302 depicting the scene 100 is provided S510. Figure 3, a second video sequence 302 includes three image frames 304a to 304c. Image frames 304a to 304c depict a foreground object 306 moving in the scene. Image frames 304a to 304c are input to an object tracker 308 for tracking the detected foreground object. There are several techniques for tracking objects in video sequences, from classical computer vision techniques to modern deep learning methods, for example, using optical flow, Kalman filtering, particle filters, correlation filters, or a deep SORT (Simple Online and Real-time Tracking with Deep Association Metric) algorithm, which uses a CNN to extract features and a Kalman filter to predict motion, etc.

[0062] The object tracker 308 may track S512 a foreground object 306 in the second video sequence 302. The tracked foreground object 306 is located in a first area 102a of a plurality of areas 102a to 102c in a scene in a first image frame 304a of the second video sequence, in a different second area 102b of a plurality of areas in the scene in a second image frame 304b, and in a different third area 102c of a plurality of areas in the scene in a third image frame 304c.

[0063] Figure 3 The system further includes a color determination component 310 , which can be used to determine the color of the tracked object 306 . Figure 2 The variability measurements 210a to 210c for each region 102a to 102c of the scene calculated in the above method can be input to the color determination component 310. The color determination component 310 can be configured to determine S514 a first set of colors for the tracked foreground object 306 in the first image frame 304a to 304c (when located in the first region 102a of the scene), a different second set of colors for the tracked foreground object 306 in the second image frame 304b (when located in the second region 102b of the scene), and a different third set of colors for the tracked foreground object 306 in the third image frame 304c (when located in the third region 102c of the scene). The different sets of colors can be determined as described above. In this way, each set of colors can include one of the following: a single color value; a plurality of color values; and a plurality of color values, each color value being associated with a probability that the tracked object has that color.

[0064] Using the variability measurements 210a to 210c, the color determination component 310 can select among the groups of colors S516. More specifically, the color determination component 310 can select a group of colors determined for the object when the object is in an area of ​​the scene with a relatively low variability measurement result. Therefore, the color determination component 310 can determine that the color 310 of the tracked object 306 is a group of colors determined for the object 306 when the object 306 is in an area of ​​the scene with a relatively low variability measurement result. In this case, the color 310 of the tracked object 306 is determined to be a group of colors determined for the object 306 in the third image frame 304c, that is, the third group of colors.

[0065] In some embodiments, the color determination component 310 can be configured to mark all occurrences of the tracked foreground object 306 in the second video sequence 302 with the determined color 310. This is particularly useful in the case of a color matching system 404 (such as the one described above in conjunction with Figures 2 to 3 The color matching system 404 may include the components 212, 214, 308, 310 discussed above, and may be configured to determine the color matching result based on the variability measurement of the region in the scene as determined using the object detected in the first video sequence (as described above in conjunction with Figure 2 The function of determining the color of the tracked object in the second video sequence (as described above in conjunction with Figure 3 The color matching system 404 may be further configured to receive a search request 408 including a first color value, determine that the first color value matches a color determined for a foreground object; and return a search response 406 based at least in part on the foreground object.

[0066] The forensic search application 402 may be configured to provide a search request 408 including a first color value to a color matching system 404 , receive a search response 406 from the color matching system 404 ; and display data from the search response to a user.

[0067] In an example, the color matching system 404 can be implemented in a single device such as a camera. In other examples, some or all of the different components (modules, units, etc.) 212, 214, 308, 310 can be implemented in a server or in the cloud. In general, the device (camera, server, etc.) that implements the components 212, 214, 308, 310 can include a circuit that is configured to implement the components 212, 214, 308, 310, and more specifically, to implement their functions. The features described in the color matching system 404 and the forensic search application 402 can be advantageously implemented in one or more computer programs that are executed on a programmable system that includes at least one programmable processor that is coupled to receive data and instructions from and transmit data and instructions to a data storage system, at least one input device (such as a camera), and in some cases at least one output device (such as a display). As an example, suitable processors for executing the instruction program include both general-purpose microprocessors and special-purpose microprocessors, as well as one or a single processor of multiple processors (or cores) of any kind of computer. The processor may be supplemented by, or incorporated in, ASICs (application specific integrated circuits).

[0068] The above embodiments should be understood as illustrative examples of the present invention. Further embodiments of the present invention are also contemplated. For example, a training phase may involve a specific time span of a day, a specific time of year, a specific weather condition, etc. When similar time periods / weather conditions occur, the variability measures determined for different time periods / weather, etc. may be, for example, Figure 3 310. In some embodiments, the variability measurements for a region of a scene are periodically reset or recalibrated.

[0069] It should be understood that any feature described with respect to any one embodiment may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other embodiment, or in combination with any combination of any other embodiment. In addition, equivalents and modifications not described above may also be employed without departing from the scope of the invention as defined in the appended claims.

Claims

1. A computer-implemented method for determining the color of a tracked object, comprising the steps of: providing a first video sequence depicting a scene, the first video sequence comprising a plurality of image frames; For each image frame of the plurality of image frames, detecting a foreground object in the image frame; for each of a plurality of regions in the scene, analyzing the first video sequence and calculating an object color probability vector associated with the region of the scene, wherein each value in the object color probability vector is associated with a color from a set of predefined colors and indicates a probability that a foreground object located in the region of the scene has the color; for each of the plurality of regions of the scene, calculating a variability measure of the probability indicated by the object color probability vector associated with the region of the scene, and associating the variability measure with the region of the scene; providing a second video sequence depicting the scene; Tracking a foreground object in the second video sequence, the tracked foreground object being located in a first area of ​​the plurality of areas in the scene in a first image frame of the second video sequence and in a different second area of ​​the plurality of areas in the scene in a second image frame of the second video sequence; determining a first set of colors for a tracked foreground object in the first image frame, and determining a second, different set of colors for a tracked foreground object in the second image frame; and determining that the color of the tracked foreground object is the first set of colors when the variability measure associated with the first area of ​​the scene is determined to be lower than the variability measure associated with the second area of ​​the scene, and otherwise determining that the color of the tracked foreground object is the second set of colors, wherein the first video sequence and the second video sequence are captured by cameras having a same field of view in a scene, wherein an area of ​​the scene corresponds to a same pixel area in the image frames of the first video sequence and the second video sequence, or wherein the first video sequence and the second video sequence are captured by a camera with a changing field of view in the scene, wherein the method further comprises the following steps: A region of pixels in an image frame from the first video sequence or the second video sequence corresponding to a region of the scene is determined using camera parameters including one or more of pan, tilt, roll, and zoom.

2. The method according to claim 1, further comprising the steps of: All occurrences of the tracked foreground object in the second video sequence are marked with the determined color.

3. The method according to claim 1, wherein: Each of the first set of colors and the second set of colors includes one of the following: A single color value; multiple color values; and A plurality of color values, each color value being associated with a probability that the tracked object has said color.

4. The method according to claim 1, wherein: Each region of the scene corresponds to a single pixel coordinate in the image frames of the first video sequence and the second video sequence.

5. The method according to claim 1, wherein: The step of detecting a foreground object in the image frame comprises: Determine a position and a range of each detected foreground object in the image frame, wherein the position and the range include one of: Pixel masks and bounding boxes.

6. The method according to claim 1, wherein: The step of analyzing the first video sequence comprises, for each foreground object located in the area of ​​the scene in an image frame of the first video sequence: determining one or more colors of the foreground object from pixel data depicting the foreground object in the image frame; as well as The determined one or more colors are used when calculating the object color probability vector associated with the region of the scene.

7. The method according to claim 1, wherein: The step of analyzing the first video sequence comprises, for each foreground object located in the area of ​​the scene in an image frame of the first video sequence: receiving a plurality of color values, each color value being associated with a probability that the foreground object in the image frame has the color; as well as The plurality of color values ​​and their associated probabilities are used when calculating the object color probability vector associated with the region of the scene.

8. The method according to claim 1, wherein: The step of calculating an object color probability vector for a region of the plurality of regions in the scene comprises detecting at least a threshold number of foreground objects in the region of the scene.

9. The method according to claim 1, wherein: The first video sequence is captured during a first time period of a day, wherein the second video sequence is captured during a second time period of a subsequent day, wherein the second time period is completely contained within the first time period.

10. The method according to claim 9, wherein: The variability measure is at least one of: Variance, standard deviation, mean absolute deviation, median absolute deviation, and coefficient of variation.

11. A non-transitory computer-readable storage medium having stored thereon instructions for implementing the method according to claim 1 when executed on a device having processing capabilities.

12. A system comprising: one or more processors; as well as One or more non-transitory computer-readable media storing first computer-executable instructions that, when executed by the one or more processors, cause the system to perform the following actions, including: providing a first video sequence depicting a scene, the first video sequence comprising a plurality of image frames; For each image frame of the plurality of image frames, detecting a foreground object in the image frame; for each of a plurality of regions in the scene, analyzing the first video sequence and calculating an object color probability vector associated with the region of the scene, wherein each value in the object color probability vector is associated with a color from a set of predefined colors and indicates a probability that a foreground object located in the region of the scene has the color; for each of the plurality of regions of the scene, calculating a variability measure of the probability indicated by the object color probability vector associated with the region of the scene, and associating the variability measure with the region of the scene; providing a second video sequence depicting the scene; Tracking a foreground object in the second video sequence, the tracked foreground object being located in a first area of ​​the plurality of areas in the scene in a first image frame of the second video sequence and in a different second area of ​​the plurality of areas in the scene in a second image frame of the second video sequence; determining a first set of colors for a tracked foreground object in the first image frame, and determining a second, different set of colors for a tracked foreground object in the second image frame; and determining that the color of the tracked foreground object is the first set of colors when the variability measure associated with the first area of ​​the scene is determined to be lower than the variability measure associated with the second area of ​​the scene, and otherwise determining that the color of the tracked foreground object is the second set of colors, wherein the first video sequence and the second video sequence are captured by cameras having a same field of view in a scene, wherein an area of ​​the scene corresponds to a same pixel area in the image frames of the first video sequence and the second video sequence, or wherein the first video sequence and the second video sequence are captured by a camera having a varying field of view of the scene, and wherein a pixel area corresponding to an area of ​​the scene in an image frame from the first video sequence or the second video sequence is determined using camera parameters, the camera parameters comprising one or more of pan, tilt, roll and zoom.

13. A system comprising a color matching system and a forensic search application, wherein: The color matching system comprises: one or more processors; and One or more non-transitory computer-readable media storing first computer-executable instructions that, when executed by the one or more processors, cause the system to perform the following actions, including: providing a first video sequence depicting a scene, the first video sequence comprising a plurality of image frames; For each image frame of the plurality of image frames, detecting a foreground object in the image frame; for each of a plurality of regions in the scene, analyzing the first video sequence and calculating an object color probability vector associated with the region of the scene, wherein each value in the object color probability vector is associated with a color from a set of predefined colors and indicates a probability that a foreground object located in the region of the scene has the color; for each of the plurality of regions of the scene, calculating a variability measure of the probability indicated by the object color probability vector associated with the region of the scene, and associating the variability measure with the region of the scene; providing a second video sequence depicting the scene; Tracking a foreground object in the second video sequence, the tracked foreground object being located in a first area of ​​the plurality of areas in the scene in a first image frame of the second video sequence and in a different second area of ​​the plurality of areas in the scene in a second image frame of the second video sequence; determining a first set of colors for a tracked foreground object in the first image frame, and determining a second, different set of colors for a tracked foreground object in the second image frame; determining that the color of the tracked foreground object is the first set of colors when it is determined that the variability measure associated with the first area of ​​the scene is lower than the variability measure associated with the second area of ​​the scene, and otherwise determining that the color of the tracked foreground object is the second set of colors; receiving a search request including a first color value; determining that the first color value matches a color determined for the foreground object; and returning a search response based at least in part on the foreground object; The forensic search application includes: one or more processors; and One or more non-transitory computer-readable media storing second computer-executable instructions that, when executed by the one or more processors, cause the forensic search application to perform actions including: providing the search request including the first color value to the color matching system; receiving a search response from the color matching system; and displaying data from the search response to a user, wherein the first video sequence and the second video sequence are captured by cameras having a same field of view in a scene, wherein an area of ​​the scene corresponds to a same pixel area in the image frames of the first video sequence and the second video sequence, or wherein the first video sequence and the second video sequence are captured by a camera having a varying field of view in the scene, and wherein a pixel area corresponding to an area of ​​the scene in an image frame from the first video sequence or the second video sequence is determined using camera parameters, the camera parameters comprising one or more of pan, tilt, roll and zoom.

Images