Method for determining the color of a tracked object

Abstract

The present disclosure relates to a method (500) for determining a color of a tracked object. Using a first video sequence and foreground objects detected therein, a color rendering metric can be determined for each of a plurality of regions in the scene. Then, in case it is determined that the tracked object has different colors in different images of a second video sequence, the color rendering metric can be used such that a color detected in a region associated with a higher color rendering metric is selected over a color detected in a region associated with a lower color rendering metric.

Description

Technical Field

[0001] This invention relates to determining the color of a tracked object in a video sequence depicting a scene, and more particularly to determining the color of a tracked object in a video sequence depicting a scene with varying lighting conditions. Background Technology

[0002] Color cast refers to the overall hue of a specific color throughout an image, usually a result of specific lighting conditions. Uneven color cast is a variation of this phenomenon, referring to different hues appearing in different areas of an image. This occurs when different light sources with different color temperatures illuminate different aspects of a scene. For example, in an outdoor environment, sunlight and street lighting may coexist, creating mixed lighting conditions. Sunlight may cast a cooler, bluer light in some parts of the scene, while street lighting may apply a warmer, yellower light in others. In addition, shadows can introduce localized color casts. The color of light in shadows is often different from that of direct light, resulting in uneven color cast.

[0003] An example of this application is in surveillance systems, particularly when operators are examining video sequences to locate objects of a specific color, where handling uneven color casts is crucial. Manually sifting through multiple video sequences to identify items of interest is tedious and time-consuming. To address this, automated video analysis tools are typically used to analyze video sequences. These tools can detect objects in the sequence and annotate detected objects using attributes such as object type, size, and speed. These features can then be used by the operator when searching for a specific object of interest (e.g., in forensic search applications). When searching for objects in an image or video sequence, an important feature is the object's color. However, the apparent color of an object can be affected by ambient light sources such as natural and artificial light sources, which can lead to a misleading impression of the object's color in an image. An example of this is a white car under a yellow streetlamp, which could cause the car to appear yellow instead of white in an image capturing the scene.

[0004] Another issue is that lighting conditions in a scene vary for many different reasons, such as the location of active artificial light sources, time of day, time of year, weather conditions, buildings, vegetation, etc. This makes it difficult to accurately determine the actual or natural color of objects captured in an image, which in turn can negatively impact the likelihood of searching for objects based on their color.

[0005] US2007 / 154088 discusses color recognition in digital images, with the aim of determining the robust perceived color or true color of an object, defined as the color of an object perceived under certain standard viewing, lighting, and sensing conditions.

[0006] CN107292933 describes the use of neural networks to determine colors.

[0007] Thierry Bouwmans et al.'s paper, "On the role and the importance of features for background modeling and foreground detection" (Arxiv.org, Cornell University Library, November 28, 2016, XP080735023), describes modeling and color variations used for detection purposes.

[0008] Therefore, improvements are needed in this area. Summary of the Invention

[0009] In view of the foregoing, it would be beneficial to address or at least reduce one or more of the aforementioned disadvantages, as described in the appended independent patent claims.

[0010] According to a first aspect of the invention, a computer-implemented method for determining the color of a tracked object is provided, comprising the steps of: providing a first video sequence depicting a scene, the first video sequence comprising a plurality of image frames; for each of the plurality of image frames, detecting a foreground object in the image frame; for each of a plurality of regions in the scene, analyzing the first video sequence and calculating an object color probability vector associated with the region of the scene, wherein each value in the object color probability vector is associated with a color from a set of predefined colors and indicates the probability that a foreground object located in the region of the scene has that color; for each of the plurality of regions in the scene, calculating a variability measurement of the probability indicated by the object color probability vector associated with the region of the scene, and associating the variability measurement with the region of the scene.

[0011] The method further includes: providing a second video sequence depicting a scene; tracking a foreground object in the second video sequence, the tracked foreground object being located in a first region of a plurality of regions in the scene in a first image frame of the second video sequence, and in a different second region of a plurality of regions in the scene in a second image frame of the second video sequence; determining a first set of colors for the tracked foreground object in the first image frame, and determining a different second set of colors for the tracked foreground object in the second image frame; and determining that the color of the tracked foreground object is the first set of colors when a variability measurement associated with the first region of the scene is lower than a variability measurement associated with the second region of the scene, otherwise determining that the color of the tracked foreground object is the second set of colors.

[0012] An "object color probability vector" is a mathematical representation associated with a specific region of a scene. Each value in this vector corresponds to a color in a set of predefined colors. This set of predefined colors may, for example, include 8 colors, 12 colors, 16 colors, etc. The granularity of this set of predefined colors often depends on the application. For example, too many shades of blue in the set may make it difficult to specify the actual blue when searching for objects, for example, in a forensic search application, and thus increase the risk of false negatives in the search. On the other hand, too few colors may lead to an increase in false positives, as all possible shades of blue that an object might have could be classified as "blue". Each value in the object color probability vector indicates the likelihood or probability that a foreground object located within a specific region of the scene possesses the corresponding color; that is, it indicates the estimated probability that a foreground object in that region of the scene will have the color corresponding to that value. It should be noted that each value in the object color probability vector can represent a probability directly or indirectly. For example, a direct representation could include the percentage of areas in the scene where an object with a "light yellow" color will appear, for example, a probability of 23%. An indirect representation could include the count of objects with the corresponding color detected in areas of the scene, for example, 15 objects with a "dark red" color detected in areas of the scene. This indirect representation needs to be compared with other values ​​in the object color probability vector to determine the actual probability of an object with the color "dark red" appearing in a region of the scene. For example, if 15 out of a total of 73 objects detected in a region of the scene have the color "dark red", the probability is 20.5%.

[0013] In the context of this specification, the term "variability measurement result" should be interpreted as a statistical term referring to how a set of data (object color probability vectors) unfolds. It indicates the degree of difference or dissimilarity within the dataset, providing the degree of "difference" between the numbers in the dataset and the mean (or center point) and between them. If all the numbers (i.e., probabilities) in the object color probability vector are close to each other and close to the mean, the variability is low. If the numbers are widely distributed, the variability is high. For example, a variability measurement result for an object color probability vector containing values ​​[0.2, 0.3, 0.3, 0.2] is lower than a variability measurement result for an object color probability vector containing values ​​[0.1, 0.7, 0.1, 0.1]. Specifically, in areas of a scene where objects present a variety of colors, the variability measurement result is generally lower. This is because multiple colors contribute equally to the probability distribution, reflecting a uniform distribution in the color spectrum. The measurement result is based on a collection of multiple objects of different colors. Therefore, when color variation is large and uniform, the associated probabilities tend to converge, resulting in lower variability because no single color significantly affects the overall measurement.

[0014] As mentioned above, the apparent color of an object will be affected by ambient light sources such as natural and artificial light sources. For example, if an area of ​​a scene is illuminated by yellow light, the apparent color of an object in that area of ​​the scene captured by the image may be shifted towards yellow, even if the object's natural color is not yellow. In another example, an area of ​​the scene is obscured by buildings. The apparent color of an object in that area of ​​the scene captured by the image may be shifted towards a darker color (e.g., dark red, dark blue, black, etc.), even if the object's natural color is not dark. For these areas in the scene, light sources and / or shadows may tilt or influence the apparent color of an object towards a particular hue (as described above, color shift), and the variability measurement will be greater for areas in the scene with more natural light. For example, if there is a yellow shift due to yellow streetlights, a white car under the light will appear yellow because the yellow light from the streetlights is reflected by the car. Similarly, a blue car will appear greener under yellow light because the car's blue color combines with the yellow light to produce green. Therefore, the probability of foreground objects (captured by the image) in a region of the scene being identified as having yellow or green may increase, while the probability of foreground objects in a region of the scene being identified as having white or blue may decrease, resulting in an increase in the variability measurement of the region of the scene under yellow streetlights compared to the region of the scene with only natural light.

[0015] The inventors have recognized that this variability measurement can be used to determine the color of a tracked object when different color groups (one or more colors) are determined for the object as it moves through a scene. During the configuration (training) phase, variability is determined by detecting foreground objects in a video sequence (or several video sequences) capturing the scene and using the image data from the video sequences to determine the colors of objects located in different regions of the scene.

[0016] As mentioned above, lower variability measurements associated with a region of the scene indicate that the colors of objects located in that region can be rendered more accurately in the captured image of the scene compared to regions associated with higher variability measurements. Therefore, variability measurements can be used as a color rendering metric associated with a specific region of the scene. The term "color rendering metric" refers to a quantitative measurement of the ability of a light source (i.e., the light source illuminating an area of ​​the scene may be affected by other objects, producing shadows, etc.) to accurately display the colors of various objects compared to ideal or natural light sources. Lower variability measurements result in higher color rendering metrics, and vice versa.

[0017] Therefore, the variability determined during the configuration phase can be used for another video sequence of the scene to determine the "actual" color of objects moving through the scene, as described herein. Advantageously, the most probable color (color group) of the object can be determined.

[0018] In some embodiments, all occurrences of the tracked foreground object in the second video sequence are marked using a determined color. Advantageously, when searching for an object of a specific color in the second video sequence, as described above, images displaying the foreground object may also be marked when it is in an area of ​​the scene where the apparent color deviates from its natural color.

[0019] In some embodiments, each of the first and second sets of colors includes one of the following: a single color value; multiple color values; and multiple color values, each associated with the probability that the tracked object has that color. For example, a neural network trained to output multiple color values ​​based on input pixel data depicting the object, each associated with the probability that the tracked object has that color, can be used to determine the set of colors. The multiple color values ​​are typically limited to a predefined set of colors. In another example, the pixel data depicting the object can be used to calculate an average color, for example, by summing all red, green, and blue values ​​from the RGB pixel data separately and then dividing by the number of pixels to obtain an average of red, green, and blue. The resulting RGB value can then be used as a single color value. In another embodiment, color quantization or clustering that groups similar colors together can be used to determine multiple color values ​​(the most common color) for the object.

[0020] When the first and second video sequences are captured by cameras with the same field of view in the scene, the regions of the scene correspond to the same pixel regions in the image frames of the first and second video sequences. The granularity of the scene regions depends on the requirements and limitations of the application. For example, fewer scene regions can result in fewer computational resources for implementing the method. On the other hand, increasing the number of regions in the scene can lead to more accurate final results (determined colors of foreground objects) because the probability of finding areas of the scene with favorable lighting can be increased. In some embodiments, each region of the scene corresponds to a single pixel coordinate in the image frames of the first and second video sequences.

[0021] In cases where the first and second video sequences are captured by a camera with changing field of view within the scene, the method further includes the step of: using camera parameters to determine a pixel region in an image frame from the first or second video sequence that corresponds to a region of the scene, the camera parameters including one or more of translation, tilt, scrolling, and zoom. Therefore, the techniques described herein can also be used for videos captured by a moving camera.

[0022] In some embodiments, the step of detecting foreground objects in an image frame includes: determining the location and extent of each detected foreground object in the image frame, wherein the location and extent include a pixel mask and a bounding box. The location and extent can be used to map the object to a specific region of the scene. If the object's bounding box / pixel mask at least partially overlaps with more than one region of the scene, the object's color can contribute to the object color probability vector associated with each of these regions.

[0023] In some embodiments, the step of analyzing the first video sequence includes, for each foreground object located in a region of a scene within an image frame of the first video sequence: determining one or more colors of the foreground object from pixel data describing the foreground object in the image frame; and using the determined one or more colors when calculating an object color probability vector associated with a region of the scene. For example, the pixel data describing the object can be used to calculate an average color, for example, by summing all red, green, and blue values ​​from RGB pixel data (or similarly for other color spaces such as HSV, CIE), and then dividing by the number of pixels to obtain an average of red, green, and blue. The resulting RGB value can then be used as a single color value. In another embodiment, color quantization or clustering that groups similar colors together can be used to determine multiple color values ​​(the most common color) for the object. The one or more colors determined for the foreground object can all be part of this group of predefined colors. Color quantization techniques can be used to map the object's color to one of the predefined colors. The determined one or more colors of the foreground object can then be used to update the object color probability vector associated with a region of the scene, for example, by updating the object count for each of the one or more colors, or by recalculating the probabilities based on the one or more colors.

[0024] In the example, the steps of analyzing the first video sequence include, for each foreground object located in a region of the scene within an image frame of the first video sequence: receiving multiple color values, each color value associated with the probability that the foreground object in the image frame has that color; and using the multiple color values ​​and their associated probabilities when computing an object color probability vector associated with a region of the scene. The multiple color values ​​can be received from a neural network trained to output multiple color values ​​based on input pixel data depicting the foreground object, each color value associated with the probability that the tracked object has that color. The multiple color values ​​are typically limited to a predefined set of colors. The object color probability vector associated with a region of the scene can then be updated using the color values ​​and their respective associated probabilities.

[0025] In some examples, the step of calculating the object color probability vector of a region in multiple regions of a scene includes detecting at least a threshold number of foreground objects in the region of the scene. In some embodiments, it is necessary to detect at least a threshold number of each of a plurality of foreground object classes. Therefore, a sufficient number of objects located in the region of the scene can be analyzed to determine the representative object color probability vector of the region of the scene. The threshold number depends on the requirements of the application and the scene captured by the video sequence. For example, the threshold number can be 50, 100, 130, 210, 450, etc.

[0026] In an embodiment, the first video sequence is captured during a first time period of a day, and the second video sequence is captured during a second time period of a subsequent day, wherein the second time period is completely contained within the first time period. Advantageously, when using object color probability vectors, the lighting conditions during the configuration / training phase of the object color probability vectors can be sufficiently well matched to the lighting conditions. The techniques described herein can be advantageously used for videos captured during the day, as this increases the likelihood that at least some areas of the scene are illuminated, making it possible for the true colors of the objects to be captured in the image. Therefore, in some embodiments, the first video sequence (and the second video sequence) are captured during the day.

[0027] In some examples, the variability measurement is at least one of variance, standard deviation, mean absolute difference, median absolute difference, and coefficient of variation.

[0028] According to a second aspect of the invention, the above objective is achieved by a non-transitory computer-readable storage medium storing instructions which, when executed on a device with processing capabilities, are used to implement the method according to the first aspect.

[0029] According to a third aspect of the invention, the above objective is achieved by a system comprising: one or more processors; and one or more non-transitory computer-readable media storing first computer-executable instructions, which, when executed by the one or more processors, cause the system to perform the actions described in detail in the appended claims.

[0030] According to a fourth aspect of the invention, as described in detail in the appended claims, the above-mentioned object is achieved by a system comprising a color matching system and a forensic search application.

[0031] Advantageously, the forensic search application can then be used to search for an object, wherein the search request includes a first color value. The first color value can be compared with a color determined for a tracked foreground object, and if it is determined that the first color value matches a color determined for the foreground object using the techniques described herein, a search response can be returned at least in part based on the tracked foreground object. The details of the actual search response can be based on the requirements of the forensic search application. For example, the search response may include finding an image of an object having a color that matches the color value of the search request, or detecting a time span of a video stream of the object, or detecting a timestamp of the object, a license plate of the object, a face of the object, etc.

[0032] The second, third, and fourth aspects may generally have the same features and advantages as the first aspect. It should also be noted that this disclosure relates to all possible combinations of features, unless otherwise expressly stated. Attached Figure Description

[0033] Figure 1 The image depicts a scene where light sources and other objects may cause uneven color shifts in the image capturing that scene.

[0034] Figure 2 The following is illustrated for calculation according to an embodiment. Figure 1 A system for measuring the variability of regions within a scene;

[0035] Figure 3 The use of the example shown is from Figure 2 A system for determining the color of the tracked foreground object based on the variability measurement results;

[0036] Figure 4 The illustration shows, according to an embodiment, including, according to Figures 2 to 3 Color matching systems and forensic search applications;

[0037] Figure 5 A flowchart illustrating a method for determining the color of a tracked object according to an embodiment is shown. Detailed Implementation

[0038] The color of an object in an image can be determined by its material and surface properties, which determine the amount of light absorbed and reflected by the object, and therefore, its perceived color. The color of an object in an image can be further affected by camera settings (white balance, color configuration, etc.) and post-processing. These properties are generally controllable by the camera owner. Properties that can be difficult to control are the light source and lighting conditions when capturing images or videos, especially in surveillance situations where the camera continuously captures a scene. Light source and lighting conditions can significantly influence the detected (apparent, perceived, etc.) color of objects in an image of a scene. This means that color can be perceived differently depending on the time of day, the time of year, weather conditions, and the object's location in the scene when the image was captured. This can cause problems when searching for objects, such as in forensic search applications based on object color, or in other applications where knowing the object's color is important.

[0039] This disclosure aims to provide methods, systems, and software for determining the color of a tracked object in a video sequence of a scene when the lighting in the scene is uneven. As described herein, uneven lighting can cause uneven color shifts in images depicting a scene, resulting in different parts of the image having different hues.

[0040] Figure 1 Scene 100 is shown to include uneven lighting. Scene 100 includes a road for vehicles 108. Therefore, scene 100 includes objects 108 moving within scene 100. Scene 100 further includes two types of natural lighting from the sun 102 and the sky, as well as artificial lighting from streetlights 106. The scene also includes trees 104 that cast shadows on a portion of scene 100. Thus, the depicted scene 100 can be divided into at least three areas 102a to 102c, each distinguished by unique lighting conditions. The first area 102a is shaded by trees. The second area 102b is illuminated by streetlights 106. Finally, the third area 102c is exposed to direct sunlight from the sun 102 and diffused light from the sky.

[0041] It should be noted that, for ease of description, simplification has been implemented. Figure 1 The scene depicted in the image, and in reality, a scene often includes more objects and / or light sources that affect the lighting conditions in different areas of the scene.

[0042] When depicted by the image / video stream capturing scene 100, a vehicle 108 moving on the road can be used to determine which area of ​​scene 100 produces the most accurate color for the object (located in that area). This will now be combined with... Figure 2 and Figure 5 Explain the calculation process of color rendering metrics for different areas of the scene.

[0043] Figure 5 A flowchart of a method 500 for determining the color of a tracked object is shown. This method includes training / configuration phases S502 to S508 and implementation phases S510 to S516. Now, in conjunction with... Figure 2 Describe the training phase.

[0044] Figure 2 The camera 202 captures images from... Figure 1 The scene, and therefore, provides a video sequence 204 depicting scene 100 in S502.

[0045] Each image frame of video sequence 204 includes a background object (background) and a foreground object 206. In this simplified example, the foreground object 206 is the vehicle 108 moving in the scene.

[0046] Object detector 212 is used to detect foreground objects 206 in each of the multiple image frames of video sequence 204 (S504 image frames). Any suitable type of object detector (both neural network-based and non-neural network-based techniques) can be used. Neural network-based techniques include those using convolutional neural networks, such as YOLO (You Only Look Once), SSD (Single-Shot Multi-Box Detector), DETR (End-to-End Detection with Transformer), and the faster R-CNN (Region-Based Convolutional Neural Network). Other neural network-based techniques include LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) networks. Non-neural network-based techniques include Histogram of Gradient Orientation (HOG), Scale Invariant Feature Transform (SIFT), and Speed-Up Robust Features (SURF).

[0047] The detected foreground object 206 is sent to the color probability calculator 214. The color probability calculator 214 can be configured to divide the scene into regions of the scene or to receive data indicating regions of the scene.

[0048] When the fixed camera 202 captures scene 100, each region of the scene corresponds to the same pixel region in the image frames of video sequence 204. For example, each region of the scene may correspond to a 10×10 pixel region, a 20×20 pixel region, a 30×20 pixel region, or any suitable pixel region in the image frames of the video sequence. In some embodiments, each region of the scene corresponds to a single pixel coordinate in the image frames of video sequence 204. In some embodiments, the region of the scene may be determined based on analysis of the scene's lighting conditions. In other embodiments, the region of the scene may be determined based on semantic segmentation of the scene's background.

[0049] In cases where the video sequence is captured by a camera whose field of view changes within the scene ( Figure 2 (Not shown in the image) Camera parameters are used to determine the pixel region in the image frames from the video sequence that corresponds to a region of the scene. Camera parameters include one or more of pan, tilt, scroll, and zoom.

[0050] Based on the above Figure 1 The description, in Figure 2 In the simplified example, the scene is captured by a fixed camera and divided into three regions 102a, 102b, and 102c.

[0051] For each region of the scene, the color probability calculator can calculate object color probability vectors 208a to 208c associated with regions 102a to 102c of the scene in S506. Each value in the object color probability vectors 208a to 208c is associated with a color from a set of predefined colors and indicates the probability that a foreground object 206 located in regions 102a to 102c of the scene has that color.

[0052] Predefined colors can be configured based on the requirements of the system described herein. For example, predefined colors may include 10 to 25 different colors or 5 to 10 colors. Increasing the number of colors may require extended training phases for the system and techniques described herein. This is because the number of thresholds for detecting foreground objects in a specific region of the scene in order to compute representative object color probability vectors 208a to 208c may be related to the number of predefined colors. On the other hand, if the number of predefined colors increases, the accuracy of determining the most suitable region for color determination in the scene (as described further below) can increase. Therefore, in some embodiments, computing object color probability vectors 208a to 208c for regions among the multiple regions 102a to 102c in scene 100 includes at least the number of thresholds for detecting foreground objects 206 in the regions of the scene. The number of thresholds may depend on the number of predefined colors, the number of moving objects 108 typically located in scene 100 at each time interval, the variability of the colors of these objects 108, etc.

[0053] The determination of the color of the foreground object 206 in the video sequence can be implemented using any suitable color determination algorithm. In some embodiments, the step of analyzing the first video sequence includes, for each foreground object located in a region of the scene in an image frame of the first video sequence: determining one or more colors of the foreground object from pixel data of the foreground object in the image frame; and using the determined one or more colors when calculating the object color probability vector associated with the region of the scene. The pixel data for a particular foreground object 206 can be determined by an object detector 212, which can determine the position and extent of each detected foreground object 206 in each image frame, wherein the position and extent include a pixel mask and a bounding box. Thus, such pixel data can be analyzed to determine one or more colors of the pixel data. For example, a color histogram can be determined and mapped to predefined colors. The predefined color of the majority of pixels in the pixel data that have the same or similar colors can be selected as the color of the object. In some cases, more than one color is determined for the object, for example, the most common color in the pixel data. Mapping the color of the pixel data to the predefined colors can include comparing the color of the pixel data with each of the predefined colors to select the “closest” color. The definition of closest match can vary based on the color space, but often, calculating the Euclidean distance between the color to be mapped and each of the predefined colors may suffice. If a sufficient number of pixels (e.g., above a certain percentage threshold) are sufficiently similar (e.g., the distance is below a certain distance threshold), it can be determined that an object has an associated predefined color. The color determined for an object located in a region of the scene can be "ticked" (an increment of value / count) in the object color probability vector of that color (associated with the region of the scene in which the object is located). After the training phase, the object color probability vector for a region of the scene might look like this (5 predefined colors, black, white, blue, red, green): [46, 120, 66, 40, 23], which can be converted to probabilities: [0.16, 0.41, 0.22, 0.13, 0.08]. For another region of the scene, these numbers may differ based on the lighting conditions described above.

[0054] In other embodiments, a neural network can be used to determine the color of an object. For example, the neural network can be trained to determine the probability that an object has a certain color. Training may include using training data, each training data set consisting of a group of pixels labeled with one or more predefined colors. The neural network can use this training data to learn how to identify the correct color (among the predefined colors) for a particular group of pixels. The output of the neural network can be multiple color values, where each color value is associated with the probability that a foreground object in an image frame has that color. For example, the output for a dark blue object might result in the following probabilities (for five predefined colors, black, white, blue, red, and green): [0.30, 0.01, 0.6, 0.05, 0.04]. This vector can be used to add the values ​​of object probability vectors 208a to 208c, or to recalculate the values ​​of object probability vectors 208a to 208c. For example, object probability vectors 208a to 208c can be normalized to a total value that always has the sum (100%) of all the individual values ​​in the vector. In some embodiments, as described above, a color among predefined colors that has a probability higher than a threshold probability of a certain input object in the output of the neural network can be "ticked" (an increase in value / count) in the color / object color probability vector of that color.

[0055] In some embodiments, an object may simultaneously occupy more than one region of a scene, for example, when the size of the object is larger than the size of a region of the scene. In this case, the determined color of an object in an image frame of a video sequence may affect more than one object probability vector. For example, as described above, the step of detecting foreground objects in an image frame includes determining the location and extent of each detected foreground object in the image frame, wherein the location and extent include one of a pixel mask and a bounding box. In such an example, all regions of the scene in which the pixel masks / bounding boxes of objects detected in the image frame at least partially overlap may be affected by the color determined for that object.

[0056] exist Figure 2 In the example, the number of predefined colors is three (3). The object probability vector 208a for region 102a (occluded by tree 104 in scene 100) is determined to be [0.1, 0.1, 0.8]. The object probability vector 208b for region 102b (illuminated by street lamp 108 in scene 100) is determined to be [0.2, 0.7, 0.1]. The object probability vector 208c for region 102c (exposed to direct sunlight from sun 102 and diffused light from the sky in scene 100) is determined to be [0.3, 0.3, 0.4].

[0057] The color probability calculator 214 is further configured to calculate, in S508, variability measurement results 210a to 210c of the probabilities indicated by the object color probability vectors associated with regions of the scene. As described above, the variability measurement results can be used as a color rendering metric, for example, a quantitative measurement of the ability of light sources in regions of the scene (i.e., light sources illuminating regions of the scene, which may be affected by other objects causing shadows, etc.) to accurately display the colors of various objects compared to ideal or natural light sources.

[0058] exist Figure 2 In this embodiment, the variability measurements 210a to 210c are calculated using the variance of the corresponding object probability vectors 208a to 208c. In other embodiments, other measurements may be used, such as standard deviation, mean absolute difference, median absolute deviation, or coefficient of variation. In any event, a lower variability measurement for a region of the scene (less variability in the probability of objects with corresponding predefined colors) indicates a region of the scene with suitable lighting for determining the "true" color of the objects, because lower variability indicates that image data of the scene captured in that region was not colored in a way that affects the accuracy of the color distribution of objects captured in that region of the scene.

[0059] exist Figure 2 In the scene, the variance 210a of the shadow region 102a is 0.109, the variance 210b of the street-lit region 102b is 0.089, and the variance 210c of the scene illuminated by natural light is 0.002. Therefore, the third region 102c of the scene is likely the most suitable region (among the three regions 102a to 102c) for determining the "true" color of the object.

[0060] Figure 3 Show Figure 2 How the metrics calculated in the middle are used to depict Figure 1 The second video sequence 302 of scene 100 in the video. Now it will be combined with Figure 5 The implementation phases S510 to S516 of the method 500 shown describe the use of the metric.

[0061] A second video sequence 302 is provided by S510 to depict scene 100. Figure 3In the second video sequence 302, three image frames 304a to 304c are included. Image frames 304a to 304c depict a foreground object 306 moving in the scene. Image frames 304a to 304c are input to an object tracker 308 for tracking the detected foreground object. Several techniques exist for tracking objects in video sequences, ranging from classical computer vision techniques to modern deep learning methods. For example, these include using optical flow, Kalman filtering, particle filters, correlation filters, or the Deep SORT (Simple Online and Real-Time Tracking with Depth Correlation Metric) algorithm, which uses CNNs to extract features and Kalman filters to predict motion.

[0062] The object tracker 308 can track a foreground object 306 in the second video sequence 302 in S512. The tracked foreground object 306 is located in a first region 102a among multiple regions 102a to 102c in the scene of the first image frame 304a of the second video sequence, in a different second region 102b among multiple regions in the scene of the second image frame 304b, and in a different third region 102c among multiple regions in the scene of the third image frame 304c.

[0063] Figure 3 The system further includes a color determination component 310, which can be used to determine the color of the tracked object 306. Figure 2 The variability measurement results 210a to 210c of each region 102a to 102c of the scene calculated in the calculation can be input to the color determination unit 310. The color determination unit 310 can be configured to determine a first set of colors for the tracked foreground object 306 in the first image frames 304a to 304c (when located in the first region 102a of the scene), a different second set of colors for the tracked foreground object 306 in the second image frame 304b (when located in the second region 102b of the scene), and a different third set of colors for the tracked foreground object 306 in the third image frame 304c (when located in the third region 102c of the scene). The different sets of colors can be determined as described above. Thus, each set of colors can include one of the following: a single color value; multiple color values; and multiple color values, each color value being associated with the probability that the tracked object has that color.

[0064] Using the variability measurement results 210a to 210c, the color determination component 310 can select from various groups of colors S516. More specifically, the color determination component 310 can select a group of colors determined for the object when it is in an area of ​​the scene with relatively low variability measurement results. Therefore, the color determination component 310 can determine that the color 310 of the tracked object 306 is a group of colors determined for the object 306 when it is in an area of ​​the scene with relatively low variability measurement results. In this case, the color 310 of the tracked object 306 is determined to be a group of colors determined for the object 306 in the third image frame 304c, i.e., the third group of colors.

[0065] In some embodiments, the color determination component 310 may be configured to mark all occurrences of the tracked foreground object 306 in the second video sequence 302 using the determined color 310. This includes a color matching system 404 (as described above). Figures 2 to 3 The color matching system 404 may be particularly advantageous in systems described above and in forensic search applications 402. The color matching system 404 may include components 212, 214, 308, 310 discussed above, and may be configured to be based on variability measurements determined using objects detected in the first video sequence, such as those of regions in the scene (as described above). Figure 2 The above) is used to achieve the function of determining the color of the tracked object in the second video sequence (as described above). Figure 3 (as described above). The color matching system 404 may be further configured to receive a search request 408 including a first color value, determine that the first color value matches a color determined for a foreground object, and return a search response 406 at least in part based on the foreground object.

[0066] The forensic search application 402 can be configured to provide a search request 408 including a first color value to the color matching system 404, receive a search response 406 from the color matching system 404, and display data from the search response to the user.

[0067] In the example, color matching system 404 can be implemented in a single device such as a camera. In other examples, some or all of the different components (modules, units, etc.) 212, 214, 308, 310 can be implemented in a server or in the cloud. Typically, the device (camera, server, etc.) implementing components 212, 214, 308, 310 can include circuitry configured to implement components 212, 214, 308, 310, and more specifically, to perform their functions. The features described in color matching system 404 and forensic search application 402 can advantageously be implemented in one or more computer programs that execute on a programmable system including at least one programmable processor coupled to receive and transmit data and instructions from a data storage system, at least one input device (such as a camera), and in some cases at least one output device (such as a display). As an example, suitable processors for executing instruction programs include both general-purpose microprocessors and special-purpose microprocessors, as well as one or a single processor from a plurality of processors (or cores) of any kind of computer. The processor can be supplemented by or incorporated into an ASIC (Application-Specific Integrated Circuit).

[0068] The above embodiments should be understood as illustrative examples of the invention. Further embodiments of the invention are also contemplated. For example, the training phase may involve a specific time span within a day, a specific time within a year, specific weather conditions, etc. When similar time periods / weather conditions occur, the variability measurement results determined for different time periods / weather conditions, etc., can be, for example, measured in... Figure 3 The color determination component 310 stores and implements the data. In some embodiments, the variability measurement results of the scene's regions are periodically reset or recalibrated.

[0069] It should be understood that any feature described with respect to any embodiment may be used alone or in combination with other described features, and may also be used in combination with one or more features of any other embodiment, or in any combination of any other embodiment. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the invention as defined in the appended claims.

Claims

1. A computer-implemented method (500) for determining the color of a tracked object, comprising the following steps: A first video sequence (204) depicting a scene (100) is provided, the first video sequence comprising a plurality of image frames; For each of the plurality of image frames, detect the foreground object in the image frame (206). For each of the multiple regions (102a to 102c) in the scene, the first video sequence is analyzed and an object color probability vector (208a to 208c) associated with the region of the scene is calculated, wherein each value in the object color probability vector is associated with a color in a set of predefined colors and indicates the probability that a foreground object located in the region of the scene has the color. For each of the plurality of regions of the scene, a variability measurement (210a to 210c) result of the probability indicated by the object color probability vector associated with the region of the scene is calculated, and the variability measurement result is associated with the region of the scene; A second video sequence (302) depicting the scene is provided. Tracking a foreground object (306) in the second video sequence, the tracked foreground object being located in a first region of the plurality of regions in the scene in the first image frame (304a) of the second video sequence, and in a different second region of the plurality of regions in the scene in the second image frame (304b) of the second video sequence; Determine a first set of colors for the tracked foreground object in the first image frame, and determine a different second set of colors for the tracked foreground object in the second image frame; and If the variability measurement result associated with the first region of the scene is determined to be lower than the variability measurement result associated with the second region of the scene, then the color (310) of the tracked foreground object is determined to be the first set of colors; otherwise, the color of the tracked foreground object is determined to be the second set of colors. Wherein, the first video sequence and the second video sequence are captured by a camera having the same field of view in the scene, wherein the region of the scene corresponds to the same pixel region in the image frames of the first video sequence and the second video sequence, or Wherein, the first video sequence and the second video sequence are captured by a camera that changes the field of view in the scene, and the method further includes the following steps: Camera parameters are used to determine the pixel region in an image frame from the first video sequence or the second video sequence that corresponds to a region of the scene. The camera parameters include one or more of pan, tilt, scroll, and zoom.

2. The method according to claim 1, further comprising the following steps: All occurrences of the tracked foreground object in the second video sequence are marked using the determined colors.

3. The method according to claim 1, wherein, Each of the first and second color groups includes one of the following: Single color value; Multiple color values; and Multiple color values, each associated with the probability that the tracked object has that color.

4. The method according to claim 1, wherein, Each region of the scene corresponds to a single pixel coordinate in the image frame of the first video sequence and the second video sequence.

5. The method according to claim 1, wherein, The steps for detecting foreground objects in the image frame include: Determine the position and extent of each detected foreground object in the image frame, wherein the position and extent include one of the following: Pixel mask and bounding box.

6. The method according to claim 1, wherein, The step of analyzing the first video sequence includes, for each foreground object in the region of the scene located in the image frame of the first video sequence: Determine one or more colors of the foreground object from pixel data depicting the foreground object in the image frame; as well as When calculating the object color probability vector associated with the region of the scene, one or more determined colors are used.

7. The method according to claim 1, wherein, The step of analyzing the first video sequence includes, for each foreground object in the region of the scene located in the image frame of the first video sequence: Receive multiple color values, each color value being associated with the probability that the foreground object in the image frame has the color; as well as When calculating the object color probability vector associated with the region of the scene, the plurality of color values ​​and their associated probabilities are used.

8. The method according to claim 1, wherein, The step of calculating the object color probability vector of a region in one of the plurality of regions in the scene includes detecting at least a threshold number of foreground objects in the region of the scene.

9. The method according to claim 1, wherein, The first video sequence is captured during a first time period of a day, and the second video sequence is captured during a second time period of a subsequent day, wherein the second time period is completely contained within the first time period.

10. The method according to claim 9, wherein, The variability measurement result is at least one of the following: Variance, standard deviation, mean absolute difference, median absolute difference, and coefficient of variation.

11. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method according to claim 1.

12. A data processing system, comprising: One or more processors; as well as One or more non-transitory computer-readable media storing first computer-executable instructions, which, when executed by the one or more processors, cause the system to perform the following actions: A first video sequence depicting a scene is provided, the first video sequence comprising a plurality of image frames; For each of the plurality of image frames, detect the foreground object in the image frame; For each of the multiple regions in the scene, the first video sequence is analyzed and an object color probability vector associated with the region in the scene is calculated, wherein each value in the object color probability vector is associated with a color from a set of predefined colors and indicates the probability that a foreground object located in the region of the scene has that color; For each of the plurality of regions of the scene, a variability measurement of the probability indicated by the object color probability vector associated with the region of the scene is calculated, and the variability measurement is associated with the region of the scene; Provide a second video sequence depicting the scene; Tracking a foreground object in the second video sequence, wherein the tracked foreground object is located in a first region of the plurality of regions in the scene in a first image frame of the second video sequence, and in a different second region of the plurality of regions in the scene in a second image frame of the second video sequence; Determine a first set of colors for the tracked foreground object in the first image frame, and determine a different second set of colors for the tracked foreground object in the second image frame; and If the variability measurement result associated with the first region of the scene is determined to be lower than the variability measurement result associated with the second region of the scene, then the color of the tracked foreground object is determined to be the first set of colors; otherwise, the color of the tracked foreground object is determined to be the second set of colors. Wherein, the first video sequence and the second video sequence are captured by a camera having the same field of view in the scene, wherein the region of the scene corresponds to the same pixel region in the image frames of the first video sequence and the second video sequence, or The first video sequence and the second video sequence are captured by a camera that changes the field of view in the scene, and wherein camera parameters are used to determine the pixel region in the image frame from the first video sequence or the second video sequence that corresponds to a region of the scene, the camera parameters including one or more of pan, tilt, scroll and zoom.

13. A data processing system comprising a color matching system (404) and an evidence search application (402), wherein, The color matching system includes: One or more processors; and One or more non-transitory computer-readable media storing first computer-executable instructions, which, when executed by the one or more processors, cause the system to perform the following actions: A first video sequence depicting a scene is provided, the first video sequence comprising a plurality of image frames; For each of the plurality of image frames, detect the foreground object in the image frame; For each of the multiple regions in the scene, the first video sequence is analyzed and an object color probability vector associated with the region in the scene is calculated, wherein each value in the object color probability vector is associated with a color from a set of predefined colors and indicates the probability that a foreground object located in the region of the scene has that color; For each of the plurality of regions of the scene, a variability measurement of the probability indicated by the object color probability vector associated with the region of the scene is calculated, and the variability measurement is associated with the region of the scene; Provide a second video sequence depicting the scene; Tracking a foreground object in the second video sequence, wherein the tracked foreground object is located in a first region of the plurality of regions in the scene in a first image frame of the second video sequence, and in a different second region of the plurality of regions in the scene in a second image frame of the second video sequence; A first set of colors for the tracked foreground object in the first image frame is determined, and a different second set of colors for the tracked foreground object in the second image frame is determined; If the variability measurement result associated with the first region of the scene is lower than the variability measurement result associated with the second region of the scene, the color of the tracked foreground object is determined to be the first set of colors; otherwise, the color of the tracked foreground object is determined to be the second set of colors. Receive a search request that includes the first color value; Determine that the first color value matches the color determined for the foreground object; and The search response is returned based at least in part on the foreground object; The forensic search application includes: One or more processors; and One or more non-transitory computer-readable media storing second computer-executable instructions, which, when executed by the one or more processors, cause the forensic search application to perform the following actions: The search request, including the first color value, is provided to the color matching system; Receive search response from the color matching system; and Display data from the search response to the user. Wherein, the first video sequence and the second video sequence are captured by a camera having the same field of view in the scene, wherein the region of the scene corresponds to the same pixel region in the image frames of the first video sequence and the second video sequence, or The first video sequence and the second video sequence are captured by a camera that changes the field of view in the scene, and camera parameters are used to determine the pixel region in the image frame from the first video sequence or the second video sequence that corresponds to a region of the scene, the camera parameters including one or more of pan, tilt, scroll and zoom.

Images