Systems and methods for managing metadata

Abstract

This invention discloses a system (300; 400), a non-transitory computer-readable storage medium (330; 430), and a method (100; 200) for managing metadata associated with a detected object in an image, wherein the object corresponds to a region of the image. A metadata item associated with the object is obtained, wherein the metadata item includes an identification of a region, and a quantitative level of an image data characteristic of the region or an encoded region is determined. A condition related to the determined quantitative level of the image data characteristic is evaluated, and if the condition is met, the metadata item is reduced or discarded. Alternatively, a compression level of an encoded region is determined, and it is determined whether the compression level is higher than a compression threshold. If the compression level is higher than the compression threshold, the metadata item is reduced or discarded.

Description

[Technical Field]

[0001] This invention relates to the management of video-related metadata, and more specifically to reducing or discarding metadata items related to a detected object in an image. [Previous Technology]

[0002] Metadata describes other data and can (for example) provide information in video data, such as information about the detected object. Metadata can (for example) make searching large amounts of stored data more efficient. Instead of searching a video stream by decoding and analyzing the image, certain information about the video stream (such as information about the detected object) can be found more easily and quickly by searching metadata.

[0003] One potential problem is that the metadata and video data generated by a camera or another type of device can be very complex and "heavy," meaning that a large amount of metadata needs to be generated, which requires transmission, bandwidth, and storage, thus requiring storage capacity. In addition, processing larger amounts of metadata can become more time-consuming.

[0004] Methods and systems for reducing the amount of metadata transmitted and / or stored in relation to object detection have been provided. However, problems remain with the transmitted and / or stored metadata, which may not be usable for subsequent analysis. Therefore, other criteria for reducing or discarding metadata are desirable. [Summary of the Invention]

[0005] One objective of the present invention is to facilitate enhanced metadata management by reducing or discarding metadata items related to detected objects in an image.

[0006] This invention is based on the following understanding: the relevance or usefulness of a metadata item of a detected object (which can form a basis for determining whether to reduce or discard metadata items) can be derived from the region corresponding to the detected object, or from the encoded image data of that region, or from the properties of that image data. The relevance or usefulness of the metadata item is related to a post-analysis or subsequent analysis of the image data depicting the image region of the detected object. For example, an identification analysis is performed on the image region to determine the identity of an object. If post-analysis cannot be performed as needed, for example, if the object is not depicted at a sufficiently high resolution for object identification, then storing metadata items about the detected object is meaningless with respect to post-analysis. This invention describes different embodiments based on this understanding.

[0007] According to a first state sample, a method is provided for managing metadata associated with a detected object in an image, wherein the object corresponds to a region of the image. The method includes: obtaining a metadata item associated with the object, wherein the metadata item includes an identification of the region; and determining a quantitative level of an image data characteristic of the region or an encoded region. The method further includes: evaluating a condition related to the determined quantitative level of the image data characteristic, and, when the condition is met, reducing or discarding the metadata item.

[0008] The term "image data characteristic of the region or encoded region" refers to a characteristic of the image data of the region or encoded region. Image data characteristics can be characteristics of image data processed or set in an image processing pipeline that encompasses one or more processing steps on the image before it is ready for encoding. However, image characteristics can also be detected in an image after encoding, and therefore, not only the quantitative level of the image data characteristics of the region but also, alternatively, the quantitative level of the image data characteristics of the encoded region can be determined and used in the method. Image data characteristics can be image data characteristics whose quantitative level affects the relevance or usefulness of a metadata item. For example, low or no usefulness or relevance of a metadata item can be attributed to a property of the region or encoded region as indicated by the quantitative level determined by the image data characteristics. Specifically, the determination of the quantitative level of the image data characteristics can indicate that no or no useful results will be produced for one or more predetermined subsequent analyses of the detected object. The characteristics of the image data and the conditions regarding the quantitative level may be selected depending on one or more predetermined follow-up analyses, such that when the conditions are met, the follow-up analysis of the detected object will not or cannot produce any useful results, and preferably, when the conditions are not met, the follow-up analysis of the detected object may or may produce useful results.

[0009] The condition relating to the assessment of the determined quantitative level refers to assessing the determined quantitative level in relation to that condition. This condition may be a quantitative condition, such as a threshold value, above or below which the determined quantitative level should satisfy the condition.

[0010] By determining a quantitative level of one of the image data characteristics of the region or the encoded region, the quantitative level of the image data characteristics determined thereby can be used to determine whether a metadata item is relevant or useful in a subsequent analysis. Specifically, the determination is made by an evaluation related to a condition used to reduce or discard a metadata item. Making this determination on whether to reduce or discard a metadata item directly based on the image data of the region of the detected object is fast and will enable the identification of metadata items that can be reduced or discarded, or otherwise would not be identified.

[0011] Discarding a metadata item means deleting the entire metadata item. Reducing metadata may include a subset of all metadata items that have been discarded.

[0012] The method according to the first state may further include maintaining the metadata item when the condition is not met. Here, maintaining may simply mean not performing a reduction or discard of the metadata item.

[0013] The action of determining the quantitative level of the image characteristics of the region or the encoded region may include determining a resolution level of the region or the encoded region, and the action of evaluating conditions related to the quantitative level of the image characteristics may include determining whether the resolution level is below a resolution threshold. The resolution threshold may be set such that when the determined resolution level is below the resolution threshold, the metadata item will have little or no use or relevance. This may be (for example) because when the resolution level is below the resolution threshold, a useful result will not or is unlikely to be produced for one or more predetermined subsequent analyses of the detected object. The predetermined subsequent analysis may be (for example) identifying or recognizing a person. A definitive identification of a person may require a minimum resolution in the image region used for identification in order to be considered legitimate. For example, a requirement may be to depict an object using at least 5 pixels per centimeter, that is, there should be at least 5 pixels along a length in the image corresponding to the natural size of the object of 1 cm.

[0014] The action of determining the quantitative level of the image characteristics of the region or the encoded region may further include determining a detail level of the object region or the encoded region, and the action of evaluating conditions related to the quantitative level of the image characteristics may further include determining whether the detail level is lower than a detail threshold. The detail threshold may be set such that when the determined detail level is lower than the detail threshold, the metadata item will have little or no use or relevance. This may (for example) be because when the detail level is lower than the detail threshold, a useful result will not or is unlikely to be produced for one or more predetermined subsequent analyses of the detected object.

[0015] The action of determining the quantitative level of the image characteristics of the region or the encoded region may further include determining the color level of one of the object regions or the encoded regions, and the action of evaluating conditions related to the quantitative level of the image characteristics may further include determining whether the color level is below a color threshold. The color threshold may be set such that when the determined color level is below the color threshold, the metadata item will have little or no use or relevance. This may (for example) be because when the color level is below the color threshold, a useful result will not or is unlikely to be produced for one or more predetermined subsequent analyses of the detected objects.

[0016] The action of determining a quantitative level of the image characteristics of the region or the encoded region may further include determining a contrast level of the region or the encoded region, and the action of evaluating conditions related to the quantitative level of the image characteristics may further include determining whether the contrast level is below a contrast threshold. The detail threshold may be set such that when the determined detail level is below the detail threshold, the metadata item will have little or no use or relevance. This may (for example) be because when the contrast level is below the contrast threshold, a useful result will not or is unlikely to be produced for one or more predetermined subsequent analyses of the detected object.

[0017] According to a second embodiment, a method is provided for managing metadata associated with a detected object in an image, wherein the object corresponds to a region of the image. The method includes: obtaining a metadata item associated with the object, wherein the metadata item includes an identification of the region; and determining a compression level of the encoded region. The method further includes: determining whether the compression level is higher than a compression threshold; and if the compression level is higher than the compression threshold, reducing or discarding the metadata item.

[0018] Unlike the method based on the first state sample, the method based on the second state sample determines a compression level of an encoded region rather than a quantitative level of an image data characteristic of the region or the encoded region. Compression is not a direct characteristic of the image data, but a characteristic of the image data processing. Specifically, the compression level indicates the degree to which the image data of the region has been compressed when the image data of the region is encoded. Furthermore, unlike image data characteristics set or processed in an image processing pipeline, the compression level is determined in an encoding section (e.g., in an encoder).

[0019] The determined compression threshold can indicate that one or more predetermined subsequent analyses of the detected object will not or are unlikely to produce any useful results. The compression threshold can be selected depending on one or more predetermined subsequent analyses, such that the condition is met when the type of the detected object or a particular predetermined subsequent analysis will not or are unlikely to produce any useful results, and preferably, the condition is not met when the type of the detected object or a particular predetermined subsequent analysis may or may produce useful results.

[0020] The compression threshold can be set such that when the determined compression level is below the compression threshold, the metadata item will have little or no use or relevance. This may be (for example) because when the determined compression level is above the compression threshold, a useful result will not or is unlikely to be produced for one or more predetermined subsequent analyses of the detected objects.

[0021] By determining a compression level of the encoded region, the determined compression level can be used to determine whether a metadata item is relevant or useful. Specifically, the determination is made by an evaluation related to a condition in the form of a compression threshold; if the value is below the compression threshold, the metadata item should be reduced or discarded. This determination of whether to reduce or discard a metadata item based on the compression level of the encoded region (i.e., indirectly based on image data) is fast and will enable the identification of metadata items that can be reduced or discarded, or otherwise would not be identified.

[0022] The method according to the second state may further include maintaining the metadata item when the compression level is equal to or lower than the compression threshold. Here, maintaining may simply mean not performing a reduction or discard of the metadata item.

[0023] The compression threshold can be set such that when the determined compression level is lower than the compression threshold, a useful result will not or cannot be produced for one or more of the detected objects in the predetermined subsequent analysis.

[0024] The action of determining the compression level may include: obtaining one or more compression values ​​of the encoded region; and determining the compression level of the encoded region based on the one or more obtained compression values. When one compression value is obtained, the compression level may be determined as that one compression value. When more than one compression value is obtained, the compression level may be determined as one of the following: the lowest value among the more than one compression level obtained, the average value among the more than one compression level obtained, the median value among the more than one compression level obtained, and the highest value among the more than one compression level obtained.

[0025] According to a third state pattern, a method for managing metadata about regions in an image is provided. The method includes performing the method according to a first state pattern or the method according to a second state pattern for each of a set of metadata items about individual objects corresponding to individual regions in the image.

[0026] According to a fourth state, a non-transitory computer-readable storage medium is provided having instructions stored thereon for implementing the method according to the first state or the method according to the second state when executed by a device having processing capabilities.

[0027] The features mentioned above regarding the methods of the first and second states also apply to this third state when applicable. To avoid undue repetition, refer to the above.

[0028] According to a fifth embodiment, a system apparatus is provided for managing metadata associated with a detected object in an image, wherein the object corresponds to a region of the image. The system includes circuitry configured to perform: a metadata acquisition function configured to acquire a metadata item associated with the object, wherein the metadata item includes an identification of the region; and a criterion determination function configured to determine a quantitative criterion of the region or an encoded image data characteristic of the region. The circuitry is further configured to perform: an evaluation function configured to evaluate a condition related to the determined quantitative criterion of the image data characteristic; and a metadata item management function configured to reduce or discard the metadata item when the condition is met.

[0029] The metadata item management function can be further configured to maintain the metadata item even if the condition is not met.

[0030] The features mentioned above in the method of the first state pattern, when applicable, also apply to this fifth state pattern. To avoid undue repetition, refer to the above content.

[0031] According to a sixth embodiment, a system is provided for managing metadata associated with a detected object in an image, wherein the object corresponds to a region of the image. The system apparatus includes circuitry configured to perform: a metadata acquisition function configured to acquire a metadata item associated with the object, wherein the metadata item includes an identification of the region; and a level determination function configured to determine a compression level of the encoded region. The system further includes: an evaluation function configured to determine whether the compression level is higher than a compression threshold; and a metadata item management function configured to reduce or discard the metadata item when the compression level is higher than the compression threshold.

[0032] The metadata item management function can be further configured to maintain the metadata item even if the condition is not met.

[0033] The features mentioned above regarding the method of the second state also apply to this sixth state when applicable. To avoid undue repetition, please refer to the above content.

[0034] Further scope of the applicability of the present invention will become apparent from the detailed description given below. However, it should be understood that while the detailed description and specific examples indicate preferred embodiments of the invention, they are given by way of illustration only, so that those skilled in the art will thereby clearly understand various changes and modifications within the scope of the invention.

[0035] Therefore, it should be understood that the present invention is not limited to specific component parts of the described apparatus or the operation of the described method, as such apparatus and methods can be varied. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be restrictive. It must be noted that, as used in this specification and the appended claims, unless the context clearly specifies otherwise, the articles "a" and "the" are intended to mean one or more of the elements present. Thus, for example, references to "a unit" or "the unit" may include several devices and the like. Furthermore, the words "comprising," "including," "containing," and similar terms do not exclude other elements or steps.

Implementation Method

[0037] The invention will now be described below with reference to the accompanying drawings, in which the presently preferred embodiments of the invention are illustrated. However, the invention may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein.

[0038] This invention applies to scenarios in which an object corresponding to a region of an image has been detected and metadata has been associated with that object. For example, this scenario occurs when tracking an object in a sequence of images in a video. The object can be detected and tracked based on motion, for example. This detection and tracking can be performed, for example, by an object detector and an object tracker in a camera, respectively. It is then beneficial to reduce or discard metadata items that are not useful or relevant (e.g., depending on one or more predetermined subsequent analyses to be performed after each image in the sequence of images in the captured, encoded, and stored video). The subsequent analysis may have identification characteristics. Not reducing or discarding such metadata items (i.e., metadata items that do not aid subsequent analysis) would result in the unnecessary transmission and / or storage of metadata items that are not useful or relevant.

[0039] Hereinafter, an embodiment of a first method 100 for managing metadata associated with a detected object in an image will be discussed with reference to FIG1. ​​The object corresponds to a region of the image. The region is typically a sub-region of the overall region of the image. The region may be defined by the boundary of the detected object or by a bounding box enclosing the detected object. The region is received as input to method 100. The method includes obtaining, in S110, a metadata item associated with the object. The metadata item associated with the object is a set of metadata about the object. The metadata item includes at least an identification of the region and metadata about one or more other types of metadata, such as an object category indication, object attributes, time, velocity (e.g., from radar), direction of movement (from a series of frames), etc. Metadata types indicating more general information may also be included in the metadata item, such as camera firmware, image frame information, such as resolution, aperture size, etc. Metadata is typically generated by the camera that captures the image, but may also include data from other sources. Regions in the image are usually identified by their coordinates. Metadata can be encapsulated (for example) in JSON, XML, or similar descriptive text formats.

[0040] Then, a quantitative level of one of the image data characteristics of region S120 or the encoded region is determined. The so-called image data characteristics of region and encoded region refer to the characteristics of image data of region and encoded region respectively. The method further includes evaluating a condition related to the determined quantitative level of image data characteristics in S130, and when the condition is met, reducing or discarding metadata items in S140.

[0041] Image data characteristics can be determined within a so-called image processing pipeline (IPP) or image pipeline of a camera. Acquiring and processing digital images is performed by an image pipeline, which is a component of any camera system. In many modern camera systems, the image pipeline is controlled by automated programs that adjust settings for processing image data within the image pipeline. An image pipeline can be considered as comprising different components or blocks that include a specific portion of their respective processing pipelines. Non-limiting examples of such components or blocks include a sensor component for image data acquisition, a gain component for applying gain, an exposure time component for controlling exposure (shutter) time, a filter component for applying filters, and a white balance component for adjusting white balance (WB).

[0042] Image characteristics can usually be detected in an image after encoding, and therefore, not only the quantitative level of the image data characteristics of the region, but also, alternatively or otherwise, the quantitative level of the image data characteristics of the encoded region can be determined and used in method 100.

[0043] Furthermore, the quantitative level of the image data characteristics of a region can be a good indicator or can be proportional to the quantitative level of the image data characteristics of the encoded region. Therefore, even if subsequent analysis is performed on an encoded image (i.e., on the encoded region), the image data characteristics of the region can be determined and used in method 100.

[0044] Reducing metadata items may include discarding certain metadata contained in a metadata item but not discarding other metadata contained in the metadata item. For example, discarding metadata of one or more metadata types but not discarding metadata of one or more other metadata types. Determining which metadata in a metadata item should be discarded or whether a metadata item should be discarded as a whole once the conditions are met can be based on the priority of one of the different types of metadata contained in the metadata item.

[0045] The substitution has a condition, which can specify a first condition and a second condition, such that if the first condition is met, the metadata item is reduced and if the second condition is met, the metadata item is discarded.

[0046] The choice between reducing or discarding metadata items may also depend on other parameters, such as the type of the detected object, the type of the planned subsequent analysis, the available capacity for transmitting images and metadata items, and the available capacity for storing images and metadata items.

[0047] Image data characteristics can typically be selected as image data characteristics whose quantitative level affects the relevance or usefulness of one of the metadata items. For example, the determined quantitative level of an image data characteristic that meets certain conditions can indicate that no or no useful results will be produced for one or more predetermined follow-up analyses of the detected object. Therefore, image data characteristics and conditions regarding quantitative levels can be selected depending on one or more predetermined follow-up analyses, such that when the conditions are met, no or no useful results will be produced for the predetermined follow-up analysis of the detected object, and preferably, when the conditions are not met, useful results may or may be produced for the follow-up analysis of the detected object. The selection can be performed in relation to a specific predetermined follow-up analysis by first selecting an image data characteristic and then setting a candidate condition for the quantitative level of the image data characteristic and then determining whether the specific predetermined follow-up analysis produces any useful results when the conditions are met. Candidate conditions can then be iteratively updated to identify a selected condition, such that when the selected condition is met, a predetermined subsequent analysis of the detected object will not or is unlikely to produce any useful results, and preferably, when the condition is not met, a subsequent analysis of the detected object may or may produce useful results. Iterative selection can also be performed for one or more selected predetermined subsequent analyses or one or more types of predetermined subsequent analyses.

[0048] The method according to the first state can further include maintaining the S150 metadata item when the condition is not met. Here, maintaining can simply mean not performing the reduction or discarding of the metadata item.

[0049] Method 100 can be performed in a system comprising more than one device, such that one or more actions (steps) are performed in one device and one or more other actions are performed in another device. Alternatively, all actions of method 100 can be performed in a single device. This device may (for example) be a monitoring camera. Furthermore, the different actions of method 100 may be performed over a short time period or over a longer time period.

[0050] For example, method 100 may be executed within a short time period in a network camera, such that the method is performed when the image has been captured by the camera, before the image is transmitted via a network to a remote location for storage and analysis. In this scenario, all actions of method 100, including reducing or discarding S140 metadata items, are preferably performed in the camera before transmitting the image and metadata items regarding the detected objects in the image, so as to reduce any metadata items before transmission. This is particularly advantageous if transmission capacity is limited.

[0051] In an alternative, method 100 can be performed over a longer time period. For example, metadata associated with the detected object can be obtained and images and related metadata can be stored for a certain time period. The determination of the quantitative level of image data characteristics S120 can then be performed at a later stage when it has been decided which subsequent analysis should be performed. Specifically, the selection of which image data characteristic is relevant can then be postponed until it has been decided which subsequent analysis should be performed. Once it has been decided which subsequent analysis should be performed and the image characteristics and conditions have been determined based on the subsequent analysis to be performed, the remainder of method 100 can be performed, thereby reducing or discarding metadata if the conditions are met. In another example, all actions except for the actual reduction or discard S140 can be performed before storing images and related metadata for a certain time period. If the conditions are met, the corresponding metadata can be marked with an indication that it should or can be reduced or discarded. Then, at a later stage, metadata is reduced or discarded from the storage device. The timing or reason for later reduction or discarding may be, for example, that a predetermined time period has elapsed since the image was captured. Reduction and discarding can be performed in series, such that metadata items are first reduced after a first predetermined time period (e.g., since storage or capture), and then discarded after a second time period since the reduction. Reduction or discarding can also be performed after metadata items have fulfilled their purpose (e.g., compilation of statistics within a certain time period, performance of all subsequent analyses, etc.), and once metadata items have served their purpose, they no longer need to be stored.

[0052] Image data characteristics may (for example) include resolution, detail, color, contrast, etc. Therefore, a quantitative standard for determining image data characteristics may refer to a quantitative standard for determining resolution, detail, color, contrast, etc. Image data characteristics are inherent properties of image data and can be directly determined from the image data itself.

[0053] The action of determining the quantitative level of the image characteristics of region S110 or the encoded region may include determining a resolution (detail) level of region or the encoded region and evaluating the conditions related to the quantitative level of the image characteristics of S130 may include determining whether the resolution (detail) level is lower than a resolution threshold.

[0054] The resolution threshold can be set such that when the determined resolution level is below the resolution threshold, the metadata item will have little or no use or relevance. This may be (for example) because when the resolution level is below the resolution threshold, a useful result will not or is unlikely to be produced for one or more predetermined subsequent analyses of the detected objects.

[0055] As an example, a predetermined subsequent analysis may depend on identifying specific features in an image of the detected object. Identifying specific features in an image of the detected object may (for example) concern identifying details of a specific size. A resolution level may then be identified below which identification would be impossible or unlikely, and therefore, the predetermined subsequent analysis would not or could not produce any useful results. On the other hand, above the identified resolution level, identification may be possible, and therefore, the predetermined subsequent analysis may produce any useful results. This resolution level may be identified based on iterative testing and using empirical results from each test, and a resolution threshold may then be set to the thus identified resolution level.

[0056] More specifically, a useful result can be determined as a valid result according to a predetermined standard. For example, valid recognition of a face may require drawing the face based on an image region using at least 2 pixels per centimeter of the drawn face. In another instance, a minimum number of horizontal pixels in the image region is required. For example, to identify a person by their face in various outdoor conditions, a minimum resolution (detail) level of 80 horizontal pixels is required in the image region depicting the face. If the resolution of the image region does not meet the minimum requirement set for a given analysis for the detection of a specific object, then the metadata related to the detection of that object is not useful to that analysis and the metadata can be reduced or discarded. Therefore, if the determined resolution (detail) level is determined to be lower than 80 horizontal pixels for a detected object in the form of a face, then a metadata related to the detected object can be reduced or discarded in subsequent analyses concerning the identification of a person.

[0057] If the resolution (detail) level is too low (e.g., due to the distance from the camera capturing the image to the detected object resulting in fewer pixels on the detected object), then more detailed analysis of objects such as mask detection, helmet detection, and accessories (backpacks, bags, weapons, etc.) will be impossible. The resolution (detail) level can be approximated (for example) by using an approximation of the natural size of the detected object. For example, if a person is detected, an approximation of a height of 1.8 m and a width between 0.5 m (front) and 0.3 m (side) can be used to determine the approximate number of pixels per centimeter of length and width in the image based on the known number of pixels per centimeter of height and width. Instead of objects whose natural size is unknown, an algorithm that takes these variables into account can be used for a given focal length and resolution at a specific distance, and the pixel density can be approximated based on knowledge of the distance between a specific point in the scene and the camera.

[0058] The lower the resolution, or in other words, the fewer pixels on the target / object, the less certain a subsequent analysis (e.g., using artificial intelligence (AI)) is regarding the classification of the object. The subsequent analysis produces a low likelihood value for the object belonging to a particular category and / or the subsequent analysis also marks the object as possibly belonging to other categories, thus making the result of the classification attempt uncertain.

[0059] The action of determining the quantitative level of the image characteristics of the region or the encoded region may further include determining one of the color levels of the region or the encoded region, and the action of evaluating the conditions related to the quantitative level of the image characteristics may further include determining whether the color level is lower than a color threshold value.

[0060] The color threshold can be set such that when the determined color level is below the color threshold, the metadata item will have little or no use or relevance. This may be (for example) because when the color level is below the color threshold, a useful result will not or is unlikely to be produced for one or more predetermined subsequent analyses of the detected objects.

[0061] As an example, a predetermined subsequent analysis may depend on identifying specific features in an image of the detected object. Identifying specific features in an image of the detected object may (for example) relate to identifying color. A color level may then be identified below which identification would be impossible or unlikely, and therefore, the predetermined subsequent analysis would not or could not produce any useful results. On the other hand, above the identified color level, identification may be possible, and therefore, the predetermined subsequent analysis may produce any useful results. This color level may be identified based on an iterative test and using empirical results from each test, and a color threshold may then be set to the thus identified color level.

[0062] For example, in the absence of color or if the color level system is low, it may be impossible to perform a specific color search for a site with high-visibility clothing requirements (e.g., classifying people wearing a yellow jacket or performing a reverse search for all human targets not wearing this color).

[0063] A specific way to identify the absence of color in an image is by means of an infrared (IR) filter mode or not in a camera, for example, related to night and day modes. If an indicator "ir_cut filter on / off" is off, the camera is in IR mode and the camera is then switched to black and white and no color will be detected.

[0064] The action of determining the quantitative level of the image characteristics of the region or the encoded region may further include determining a contrast level of the region or the encoded region, and the action of evaluating the conditions related to the quantitative level of the image characteristics may further include determining whether the contrast level is lower than a contrast threshold.

[0065] The contrast threshold can be set such that when the determined contrast level is below the contrast threshold, the metadata item will have little or no use or relevance. This may be (for example) because when the contrast level is below the contrast threshold, a useful result will not or is unlikely to be produced for one or more predetermined subsequent analyses of the detected objects.

[0066] As an example, a predetermined subsequent analysis may depend on identifying specific features in an image of the detected object. Identifying specific features in an image of the detected object may (for example) concern identifying details of a specific size. A contrast level may then be identified below which such identification would be impossible or unlikely, and therefore the predetermined subsequent analysis would not or could not produce any useful results. On the other hand, above the identified contrast level, such identification may be possible, and therefore the predetermined subsequent analysis may produce any useful results. This contrast level may be identified based on repeated testing and using empirical results from each test, and a contrast threshold may then be set to the thus identified contrast level.

[0067] The contrast level can be determined based on a known algorithm. As an example, the contrast can be calculated based on the relative illumination between pixels in a region or an encoded region.

[0068] The lower the contrast, the less certain a subsequent analysis (e.g., using artificial intelligence (AI)) is regarding the classification of an object. This subsequent analysis produces a low likelihood value for the object belonging to a particular category and / or also indicates that the object may belong to other categories, thus making the classification attempt uncertain. If there is no contrast or low contrast, it will be difficult or impossible to find an anchor point to summarize anything in a subsequent analysis.

[0069] Hereinafter, an embodiment of a second method 200 for managing metadata associated with a detected object in an image will be discussed with reference to FIG2. The object corresponds to a region of the image. This region is typically a sub-region of the overall region of the image. The region may be defined by the boundary of the detected object or by a bounding box enclosing the detected object. The region is received as input to method 200. The method includes obtaining, in S210, a metadata item associated with the object. The metadata item associated with the object is a set of metadata about the object. The metadata item includes at least an identification of the region and metadata about one or more other types of metadata, such as an object category indication, object attributes, time, velocity (e.g., from radar), direction of movement (from a series of frames), etc. Metadata types indicating more general information may also be included in the metadata item, such as camera firmware, image frame information, such as resolution, aperture size, etc. Metadata is typically generated by the camera that captures the image, but may also include data from other sources. Regions in the image are usually identified by their coordinates. Metadata can be encapsulated (for example) in JSON, XML, or similar descriptive text formats.

[0070] Determine the compression level of one of the encoded regions S220. The compression level of the encoded region refers to the compression level of the encoded image data of the encoded region in relation to the image data of the region before encoding. The method further includes determining S230 whether the compression level is higher than a compression threshold, and if the compression level is higher than the compression threshold, reducing or discarding S240 metadata items.

[0071] Instead of using the compression level itself, any result of the coding analysis that leads to the determination of the compression value can be used to determine whether metadata items should be reduced or discarded. If the coding analysis concludes that a high compression value will be used to encode the region of the image corresponding to the object, then metadata items should have been reduced or discarded based on this conclusion, because the detail level is likely to be reduced, so that the transmitted and stored image will not contain any relevant information anyway. In addition, the compression setting may include regions that have been defined as always being less relevant and therefore highly compressed. For example, a user may define the sky (or the upper part of the image) as less relevant, or certain scene analyses (semantic segmentation) may have identified less relevant regions. All metadata items concerning objects detected in such highly compressed regions can be reduced or discarded.

[0072] Reducing metadata items may include discarding certain metadata contained in a metadata item but not discarding other metadata contained in the metadata item. For example, discarding metadata of one or more metadata types but not discarding metadata of one or more other metadata types. Determining which metadata in a metadata item should be discarded or whether a metadata item should be discarded as a whole once the conditions are met can be based on the priority of one of the different types of metadata contained in the metadata item.

[0073] The substitution has a condition, which can specify a first condition and a second condition, such that if the first condition is met, the metadata item is reduced and if the second condition is met, the metadata item is discarded.

[0074] The choice between reducing or discarding metadata items may also depend on other parameters, such as the type of the detected object, the type of the planned subsequent analysis, the available capacity for transmitting images and metadata items, and the available capacity for storing images and metadata items.

[0075] Unlike method 100 described with respect to FIG. 1, method 200 determines a compression level of an encoded region rather than a quantitative level of an image data characteristic of the region or the encoded region. The compression level is not a direct characteristic of the image data, but rather a characteristic of the image data processing. Specifically, the compression level indicates the degree to which the image data of the region has been compressed when the image data of the region is encoded. Furthermore, unlike image data characteristics set or processed in an image processing pipeline, the compression level is determined in an encoding section (e.g., in an encoder). Encoding standards such as H.264 and H.265 can be used as compression schemes by the encoder.

[0076] The compression threshold can be set such that when the determined compression level is higher than the compression threshold, the metadata item will have little or no use or relevance. This may be (for example) because when the compression level is higher than the compression threshold, a useful result will not or is unlikely to be produced for one or more predetermined subsequent analyses of the detected objects.

[0077] As an example, a predetermined subsequent analysis may depend on identifying specific features in an image of the detected object. Identifying specific features in an image of the detected object may (for example) concern identifying details of a specific size. A compression level may then be identified below which such identification would be impossible or unlikely, and therefore, the predetermined subsequent analysis would not or could not produce any useful results. On the other hand, above the identified compression level, such identification may be possible, and therefore, the predetermined subsequent analysis may produce any useful results. This compression level may be identified based on an iterative test and using empirical results from each test, and a compression threshold may then be set to the thus identified compression level.

[0078] Method 200 may further include maintaining the S250 metadata item when the compression level is equal to or below the compression threshold. Here, maintaining may simply mean not performing a reduction or discarding of the metadata item.

[0079] The action of determining the compression level of S220 may include obtaining one or more compression values ​​of the encoded region, and determining the compression level of the encoded region based on one or more of the obtained compression values.

[0080] For an image, a compression value can be provided for each pixel block (for example, 16x16 pixels or 64x64 pixels) of the image's pixel blocks. In this case, one or more pixel blocks included in the coded region corresponding to the detected object will be used to determine the compression level of the coded region. When the coded region corresponds to one pixel block, a compression value is obtained, and the compression level can be determined as the obtained compression value. When the coded region corresponds to more than one pixel block, more than one compression value is obtained, that is, the compression value of the pixel blocks of the coded region. The compression level can then be determined as one of the following: the lowest value among the more than one compression value obtained, the average value among the more than one compression value obtained, the median value among the more than one compression value obtained, and the highest value among the more than one compression value obtained.

[0081] As another alternative, metadata items may be reduced or discarded if at least a certain percentage of the encoded region is associated with a compression value higher than the compression threshold.

[0082] The compressed value can be obtained (for example) in the form of a quantization parameter. The quantization parameter for different parts of the image can be obtained in the form of a quantization parameter graph.

[0083] Method 200 can be performed in a system comprising more than one device, such that one or more actions (steps) are performed in one device and one or more other actions are performed in another device. Alternatively, all actions of method 200 can be performed in a single device. This device may (for example) be a video camera. Furthermore, the different actions of method 200 may be performed over a short time period or over a longer time period.

[0084] For example, method 200 may be executed within a short time period in a network camera, such that the method is performed when the image has been captured by the camera, before the image is transmitted via a network to a remote location for storage and analysis. In this scenario, all actions of method 200, including reducing or discarding S240 metadata items, are preferably performed in the camera before transmitting the image and metadata items regarding the detected objects in the image, so as to reduce any metadata items before transmission. This is particularly advantageous if transmission capacity is limited.

[0085] In an alternative, method 200 can be performed over a longer time period. For example, metadata associated with the detected object can be obtained and images and related metadata can be stored for a certain time period. The determination of the compression level S220 can then be performed at a later stage when it has been decided which subsequent analysis should be performed. Specifically, the selection of the compression threshold can then be postponed until it has been decided which subsequent analysis should be performed. Once it has been decided which subsequent analysis should be performed and the compression threshold has been determined based on the subsequent analysis to be performed, the remainder of method 200 can be performed, thereby causing the metadata to be reduced or discarded if the conditions are met. In another example, all actions except for the actual reduction or discard S240 can be performed before the images and related metadata are stored for a certain time period. If the conditions are met, the corresponding metadata can be marked with an indication that it should or can be reduced or discarded. Then, at a later stage, the metadata is reduced or discarded from the storage device. The timing or reason for later reduction or discarding may be, for example, that a predetermined time period has elapsed since the image was captured. Reduction and discarding can be performed in series, such that metadata items are first reduced after a first predetermined time period (e.g., since storage or capture), and then discarded after a second time period since the reduction. Reduction or discarding can also be performed after metadata items have fulfilled their purpose (e.g., compilation of statistics within a certain time period, performance of all subsequent analyses, etc.), and once metadata items have served their purpose, they no longer need to be stored.

[0086] Methods 100 and 200 may be repeatedly performed for each of a set of metadata items relating to individual objects, which correspond to individual regions in the image.

[0087] Figure 3 illustrates a schematic diagram relating to an embodiment of a first system 300 of the present invention, which manages metadata associated with a detected object in an image, wherein the object corresponds to a region of the image. System 300 includes circuitry 310. Circuitry 310 is configured to perform the functions of system 300. Circuitry 310 may include a processor 312, such as a central processing unit (CPU), microcontroller, or microprocessor. Processor 312 is configured to execute program code. The program code may (for example) be configured to perform the functions of system 300.

[0088] System 300 may further include a memory 330. The memory 330 may be one or more of the following: a buffer, a flash memory, a hard disk drive, a removable media, a volatile memory, a non-volatile memory, a random access memory (RAM), or another suitable device. In a typical configuration, the memory 330 may include a non-volatile memory for long-term data storage and a volatile memory used as system memory for circuit 310. The memory 330 may exchange data with circuit 310 via a data bus. Accompanying control lines and an address bus may also be present between the memory 330 and circuit 310.

[0089] The functions of system 300 may be embodied in the form of executable logic constants (e.g., code areas, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory 330) of system 300 and executed by circuit 310 (e.g., using processor 312). Furthermore, the functions of system 300 may be a standalone software application or part of a software application that performs additional tasks related to system 300. The described functions may be considered as a method of execution by a processing unit (e.g., processor 312 of circuit 310). Moreover, although the described functions may be implemented in software, these functionalities may also be implemented via dedicated hardware or firmware, or a combination of hardware, firmware, and / or software.

[0090] Circuit 310 is configured to perform: a meta-data acquisition function 331, which is configured to acquire a meta-data item associated with an object, wherein the meta-data item includes an identification of a region; and a bit-level determination function 332, which is configured to determine a compressed bit level of an encoded region.

[0091] Circuit 310 is further configured to perform a level determination function 332, which is configured to determine a quantitative level of one of the characteristics of image data of a region or an encoded region.

[0092] Circuit 310 is further configured to perform an evaluation function 333, which is configured to evaluate a condition related to a quantitative level determined by the characteristics of the image data.

[0093] Circuit 310 is further configured to execute a meta data item management function 334, which is configured to reduce or discard meta data items when conditions are met.

[0094] Meta data item management function 334 can be further configured to maintain meta data items when conditions are not met.

[0095] System 300 can be implemented in more than one device, such that one or more of the functions are implemented in one device and one or more other functions are implemented in another device. Alternatively, all the functions of system 300 can be implemented in a single device. The single device may be, for example, a surveillance camera. To further illustrate this, reference is made to the description of method 100 with reference to FIG1.

[0096] The function performed by circuit 310 can be further adapted to the corresponding steps of an embodiment of the method described in FIG1.

[0097] Figure 4 illustrates a schematic diagram relating to an embodiment of a second system 400 of the present invention, which manages metadata associated with a detected object in an image, wherein the object corresponds to a region of the image. System 400 includes circuitry 410. Circuitry 410 is configured to perform the functions of system 400. Circuitry 410 may include a processor 412, such as a central processing unit (CPU), microcontroller, or microprocessor. Processor 412 is configured to execute program code. The program code may, for example, be configured to perform the functions of system 400.

[0098] System 400 may further include a memory 430. Memory 430 may be one or more of the following: a buffer, a flash memory, a hard disk drive, a removable media, a volatile memory, a non-volatile memory, a random access memory (RAM), or another suitable device. In a typical configuration, memory 430 may include a non-volatile memory for long-term data storage and a volatile memory for use as system memory in circuit 410. Memory 430 may exchange data with circuit 410 via a data bus. Accompanying control lines and an address bus may also be present between memory 430 and circuit 410.

[0099] The functions of system 400 may be embodied in the form of executable logic constants (e.g., code areas, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory 430) of system 400 and executed by circuit 410 (e.g., using processor 412). Furthermore, the functions of system 400 may be a standalone software application or part of a software application that performs additional tasks related to system 400. The described functions may be considered as a method configured to execute by a processing unit (e.g., processor 412 of circuit 410). Moreover, although the described functions may be implemented in software, these functionalities may also be implemented via dedicated hardware or firmware, or a combination of hardware, firmware, and / or software.

[0100] Circuit 410 is configured to perform a metadata acquisition function 431, which is configured to acquire a metadata item associated with an object, wherein the metadata item includes an identification of a region.

[0101] Circuit 410 is further configured to perform a level determination function 432, which is configured to determine one of the compressed levels of the encoded region.

[0102] Circuit 410 is further configured to perform an evaluation function 433, which is configured to determine whether the compression level is higher than a compression threshold.

[0103] Circuit 410 is further configured to execute a meta data item management function 434, which is configured to reduce or discard meta data items when the compression level is higher than the compression threshold.

[0104] The metadata item management function 334 can be further configured to maintain metadata items when the compression level is lower than or equal to the compression threshold.

[0105] System 400 can be implemented in more than one device, such that one or more of the functions are implemented in one device and one or more other functions are implemented in another device. Alternatively, all the functions of system 400 can be implemented in a single device. The single device may be, for example, a camera. To further illustrate this, reference is made to a description of method 100 with reference to FIG1.

[0106] The function performed by circuit 410 can be further adapted to the corresponding steps of the embodiment of the method described in FIG2.

[0107] Those skilled in the art will recognize that the present invention is not limited to the embodiments described above. Rather, many modifications and variations are possible within the scope of the appended claims. Based on a study of the drawings, disclosures, and the appended claims, those skilled in the art will understand and implement such modifications and variations in practicing the claimed invention. [Simplified Explanation of the Diagram]

[0036] The above and other aspects of the invention will now be described in more detail with reference to the accompanying drawings. These drawings should not be considered limiting but are for explanation and understanding. Figure 1 shows a flowchart relating to an embodiment of a first method of the invention, which is used to manage metadata associated with a detected object in an image. Figure 2 shows a flowchart relating to an embodiment of a second method of the invention, which is used to manage metadata associated with a detected object in an image. Figure 3 shows a schematic diagram relating to an embodiment of a first system of the invention, which is used to manage metadata associated with a detected object in an image. Figure 4 shows a schematic diagram relating to an embodiment of a system of the invention, which is used to manage metadata associated with a detected object in an image.

Claims

1. A method for managing metadata associated with one or more detected objects in an image, wherein each of the one or more objects corresponds to a separate region of the image, the method comprising: Obtain a metadata item associated with one or more of the objects, wherein the metadata item includes an identification of one of the regions; determine the resolution level of the region or an encoded region; determine whether the resolution level is lower than a resolution threshold, wherein the resolution threshold is set such that when the determined resolution level is lower than the resolution threshold, no useful results will be produced or impossible to produce for the predetermined subsequent analysis of the detected objects one or more; and when the resolution level is lower than the resolution threshold, reduce or discard the metadata item and retain the image containing the one or more objects.

2. The method of request item 1, further comprising: If the resolution level is equal to or higher than the resolution threshold, the metadata item is maintained.

3. A method for managing metadata associated with one or more detected objects in an image, wherein each of the one or more objects corresponds to a separate region of the image, the method comprising: Obtain a metadata item associated with one or more of the objects, wherein the metadata item includes an identification of one of the regions; determine the contrast level of the region or an encoded region; determine whether the contrast level is lower than a contrast threshold, wherein the contrast threshold is set such that when the determined contrast level is lower than the contrast threshold, no useful results will be produced or impossible to produce for the predetermined subsequent analysis of the detected objects one or more; and when the contrast level is lower than the contrast threshold, reduce or discard the metadata item and retain the image containing the one or more objects.

4. The method of claim 3, further comprising: When the contrast level is equal to or higher than the contrast threshold, the metadata item is maintained.

5. A method for managing metadata associated with one detected object in an image, wherein the object corresponds to a region of the image, the method comprising: Obtain a metadata item associated with the object, wherein the metadata item includes an identification of one of the regions; determine a compression level of one of the encoded regions; determine whether the compression level is higher than a compression threshold, wherein the compression threshold is set such that when the compression level is higher than the compression threshold, no useful results will be produced or impossible to be produced for one or more predetermined subsequent analyses of the detected object; and reduce or discard the metadata item when the compression level is higher than the compression threshold.

6. The method of claim 5, further comprising: When the compression level is equal to or lower than the compression threshold, the metadata item is maintained.

7. The method of request item 5, wherein determining the compression level includes: Obtain one or more compressed values ​​for the encoded region; And based on the one or more obtained compression values, determine the compression level of the encoded region.

8. As in request item 7, wherein, When a compression value is obtained, the compression level is determined as that compression value, and when more than one compression value is obtained, the compression level is determined as one of the following: the lowest value among the more than one compression level, the average value among the more than one compression level, the median value among the more than one compression level, and the highest value among the more than one compression level.

9. A method for managing metadata about a region in an image, the method comprising: Perform the method as requested in claim 1 for each of a set of metadata items relating to individual objects corresponding to individual regions in the image.

10. A non-transitory computer-readable storage medium having instructions stored thereon for implementing any one of claims 1 to 9 when executed by a device having processing capabilities.

11. A system for managing metadata associated with one or more detected objects in an image, wherein each of the one or more objects corresponds to a separate region of the image, the system comprising circuitry configured to perform: a metadata acquisition function configured to acquire a metadata item associated with one of the one or more objects, wherein the metadata item includes an identification of the region; a criterion determination function configured to determine a resolution level of the region or an encoded region; an evaluation function configured to determine whether the resolution level is below a resolution threshold, wherein the resolution threshold is set such that when the determined resolution level is below the resolution threshold, no useful results will be produced or it is unlikely to produce any results for any predetermined subsequent analysis of the one or more detected objects; and a metadata item management function configured to reduce or discard the metadata item and retain the image containing the one or more objects when the resolution level is below the resolution threshold.

12. A system for managing metadata associated with one or more detected objects in an image, wherein each of the one or more objects corresponds to a separate region of the image, the system comprising circuitry configured to perform: a metadata acquisition function configured to acquire a metadata item associated with one of the one or more objects, wherein the metadata item includes an identification of the region; a threshold determination function configured to determine a contrast level of the region or an encoded region; an evaluation function configured to determine whether the contrast level is below a contrast threshold, wherein the contrast threshold is set such that when the determined contrast level is below the contrast threshold, no useful results will be produced or impossible to be produced for any predetermined subsequent analysis of the one or more detected objects; and a metadata item management function configured to reduce or discard the metadata item and retain the image containing the one or more objects when the contrast level is below the contrast threshold.

13. A system for managing metadata associated with a detected object in an image, wherein the object corresponds to a region of the image, the system comprising circuitry configured to perform: a metadata acquisition function configured to acquire a metadata item associated with the object, wherein the metadata item includes an identification of the region; a level determination function configured to determine a compression level of the encoded region; an evaluation function configured to determine whether the compression level is higher than a compression threshold, wherein the compression threshold is set such that when the compression level is higher than the compression threshold, no useful results will be produced or it is unlikely to produce any results for one or more predetermined subsequent analyses of the detected object; and a metadata item management function configured to reduce or discard the metadata item when the compression level is higher than the compression threshold.

14. The system of request item 13, wherein the level determination function is configured to: obtain one or more compressed values ​​of the encoded region; and determine the compression level of the region based on the one or more obtained compressed values.

15. The system of request item 14, wherein the level determination function is configured to: when a compression value is obtained, determine the compression level as the one compression value, and when more than one compression value is obtained, determine the compression level as one of the following: the lowest value among the more than one compression value, the average value among the more than one compression value, the median value among the more than one compression value, and the highest value among the more than one compression value.

Images