Improved object attribute classification

Abstract

To provide a method for classifying object attributes in an image.SOLUTION: A method includes: obtaining a plurality of object proposals from an artificial neural network entity trained to localize and classify objects using a plurality of feature map layers associated with different spatial resolutions (S410); identifying a main object proposal and one or more other object proposals (S420); ranking the feature map layers from a least significant feature map layer to a most significant feature map layer (S430); and determining an attribute class for a first attribute based on attribute class confidence scores of the main and other proposals (S440). The determination of the attribute class for the first attribute includes taking the ranking of the feature map layers as well as object location overlaps into account.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] This disclosure generally relates to the field of image processing. More specifically, this disclosure relates to the localization and classification of objects in images, and to the classification of object attributes. [Background technology]

[0002] Artificial neural network (ANN)-based architectures have proven useful for detecting instances of (semantic) objects of certain object classes in images, such as still images or video frames. Examples of object classes include, for example, humans, animals, vehicles, license plates, faces, and so on. In addition to simply determining whether an object of a certain object class may be present in an image, such architectures can also be trained to estimate where that object is located within the image. A modern example of such an architecture is the feature pyramid network (FPN), which is considered particularly well-suited for detecting objects at different scales.

[0003] ANN-based architectures can also be used to classify one or more attributes, features, or even activities of an object. Object attributes may include, for example, color, texture, object shape, and context, such as whether the object is currently facing a certain direction, whether the object is moving or stationary, whether the object is positioned on a particular type of surface (such as a road or sidewalk), or any other details of the object or its location that can be categorized and labeled. Attribute classification is considered an important task, for example, in computer vision, because proper classification of attributes can help distinguish between different objects present in the same scene.

[0004] However, modern ANN-based architectures are often unable to distinguish between different objects and instead work by providing several object proposals. Each such proposal may include, for example, the proposed object location, confidence scores for one or more object classes, and confidence scores for one or more attribute classes for one or more attributes. Therefore, post-processing of the output from an ANN-based architecture is required to determine, for example, how many distinct objects there are in an image, where those objects are located, and the most likely objects and / or attribute classes for those objects. Known post-processing operations include the use of so-called non-maximum suppression (NMS) and intersection-over-union (IOU) measures.

[0005] This disclosure aims to advance such post-processing of object proposals from ANN-based architectures and mitigate one or more shortcomings of modern technologies. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Dong Liu et al., Simultaneous object localization and attribute classification using multitask deep neural networks, US11087130B2 [Non-patent literature]

[0007] [Non-Patent Document 1] Kaiming He et al., Deep Residual Learning for Image Recognition, https: / / doi.org / 10.48550 / arXiv.1512.03385 [Non-Patent Document 2] Jing Xu et al., RegNet: Self-Regulated Network for Image Classification, https: / / doi.org / 10.48550 / arXiv.2101.00590 [Non-Patent Document 3] Tsung-Yi Lin et al., Feature Pyramid Networks for Object Detection, https: / / doi.org / 10.48550 / arXiv.1612.03144 [Non-Patent Document 4] Mingxing Tan et al., EfficientDet: Scalable and Efficient Object Detection, https: / / doi.org / 10.48550 / arXiv.1911.09070 [Non-Patent Document 5] Wei Liu et al., SSD:Single Shot MultiBox Detector, https: / / doi.org / 10.48550 / arXiv.1512.02325 [Non-Patent Document 6] Tsung-Yi Lin et al., Focal Loss for Dense Object Detection, https: / / doi.org / 10.48550 / arXiv.1708.02002 [Non-Patent Document 7] Takuya Akiba et al., Optuna:A Next-generation Hyperparameter Optimization Framework, https: / / doi.org / 10.48550 / arXiv.1907.10902 [Overview of the project]

[0008] For the purposes set forth above, this disclosure proposes improved methods, devices, computer programs, and computer program products for object attribute classification in images, as defined by the attached independent claims. Various embodiments are defined by the attached dependent claims.

[0009] A first aspect of this disclosure provides a method for classifying object attributes in an image. The method involves obtaining a plurality of object proposals in the same image from the output of an artificial neural network (ANN) entity trained to locate and classify objects in an image using a plurality of feature map layers associated with different spatial resolutions, wherein each object proposal represents at least i) an object class confidence score for the same first object class, ii) an estimated object location in the image, and iii) an attribute class confidence score for each of one or more different attribute classes for the same first attribute, and each object proposal is associated with one of the feature map layers, thereby obtaining a plurality of object proposals in the same image. The method further includes identifying a first set of object proposals, including a primary object proposal and one or more other object proposals, from among the plurality of object proposals and based on their respective indicated object class confidence scores. The method further includes ranking the feature map layers from least significant to most significant. The method determines an attribute class for a first attribute based on one or more attribute class confidence scores of all members of a first set, further comprising determining the attribute class for the first attribute, which as part of the determination includes ranking feature map layers related to one or more other object proposals, and taking into account object location overlaps between the main object proposal and one or more other object proposals.

[0010] As will be described in more detail later in this specification, the proposed method improves upon currently available techniques by not simply ignoring information relating to object attributes found in one or more object proposals that are not considered the primary object proposal. Instead, the proposed method uses this information together with the information provided by the primary object proposal, and in particular, when determining the final estimated attribute class for a particular first attribute of an object, it also considers whether the information originating from a particular feature map layer should be given a greater or lesser weight. As will be illustrated later in this specification, this reduces the risk of misclassifying attributes, for example, when two or more objects are present in the same part of an image.

[0011] In one or more embodiments of this method, the object location of each object proposal may be represented as a bounding box, and the method may further include using an Intersection / Union (IOU) measure / behavior to determine object location overlaps between the principal object proposal and one or more other object proposals. Such measures are generally known, readily available, and can be efficiently implemented on modern hardware.

[0012] In one or more embodiments of the method, taking into account ranking and object location overlap may include the fact that an attribute class confidence score indicated by an object proposal, associated with a feature map layer that has greater overlap with the primary object proposal and is ranked as more important, is more important for determining the attribute class of a first attribute than an attribute class confidence score indicated by an object proposal, associated with a feature map layer that has less object overlap with the primary object proposal and is ranked as less important. In other words, the assumed method may include the fact that, in order for the information of a particular object proposal to be considered more relevant for determining the attribute class of a first attribute, this object proposal should arise from a feature map layer that has sufficiently large overlap (with respect to the estimated object location) with the primary object proposal and is ranked as more relevant (i.e., better at determining the first attribute of an object) for determining the first attribute of an object. This is compared, for example, to any other object proposals arising from feature map layers that have less overlap with the main object proposal and are ranked as less relevant to determining the object's first attribute (i.e., worse at determining the object's first attribute). For example, if a particular feature map layer is considered better at determining the object's color (i.e., operates at a resolution more suitable for determining the object's color), then the information from object proposals arising from this particular feature map layer will be given more weight in determining the attribute class for the first attribute, insofar as these object proposals also propose locations that are sufficiently similar to the locations proposed by the main object proposal.

[0013] In one or more embodiments of the Method, determining an attribute class for a first attribute may further include iterating over a first set of members, each of which represents an attribute class confidence score for a particular attribute class. For each iteration, determine a term equal to or proportional to the product of the object location overlap size between the members of the first set and the principal object proposal, the overall ranking score for the feature map layer associated with the member, and the attribute class confidence score for the particular attribute class represented by the member; and determine the overall attribute class score for the particular attribute class as equal to or proportional to the sum of the terms determined during the iteration. The Method may further include determining the attribute class for the first attribute as the attribute class having the highest overall attribute class score. Such a procedure may help implement the above-described concept of assigning higher importance to object proposals that closely overlap with the principal object proposal (with respect to object location) and that originate from feature map layers that are considered better and more relevant for classifying the first attribute.

[0014] In one or more embodiments of this method, iteration may be performed only over a first set of members that exhibit the highest attribute class confidence score for a particular attribute class. In other words, it is assumed that the iteration is performed only over the first set of members for which a particular attribute class has the top 1 attribute class confidence score. In other examples, it may instead be iterated over all members of the first set, regardless of whether a particular attribute class has the top 1 attribute class confidence score.

[0015] In one or more embodiments of this method, the overall attribute class score for each particular attribute class may be further defined as being inversely proportional to the number of members in the first set that is iterated over. For example, if it is iterated over a total of P members, the overall attribute class score may be scaled by a factor of 1 / P, or similar.

[0016] In one or more embodiments of this method, the overall ranking score for a feature map layer associated with a member may be defined as the ratio of the ranking score for the feature map layer to the sum of such ranking scores for all of the feature map layers. For example, each l-th feature map layer has a ranking score w l If assigned, the overall ranking score for the l-th feature map layer is: It can be defined as TIFF0007867528000001.tif5170.

[0017] In one or more embodiments of this method, the ranking of multiple feature map layers may depend on a first attribute. In other words, the ranking of feature map layers may differ for different attributes, which helps to take into account that different feature map layers may be more or less relevant (i.e., better or worse) for classifying a particular type of attribute.

[0018] In one or more embodiments of the Method, the Method may further include using non-maximal suppression (NMS) behavior to identify the primary object proposal and one or more other object proposals.

[0019] In one or more embodiments of this method, the ANN entity may include a feature pyramid network (FPN) for providing multiple feature map layers. As mentioned above, FPNs can be particularly useful for detecting objects at different (spatial) scales, such as both larger and smaller objects in the same image.

[0020] In one or more embodiments of this method, the ANN entity comprises multiple convolutional layers to provide different spatial resolutions.

[0021] In one or more embodiments of this method, the method is carried out in / by a surveillance camera.

[0022] A second aspect of this disclosure provides a device for classifying object attributes in an image. The device includes a processing circuit (such as a processor) which obtains a plurality of object proposals in the same image from the output of an artificial neural network (ANN) entity trained to locate and classify objects in an image using a plurality of feature map layers associated with different spatial resolutions, each object proposal representing at least i) an object class confidence score for the same first object class, ii) an estimated object location in the image, and iii) an attribute class confidence score for each of one or more different attribute classes for the same first attribute, and each object proposal represents a plurality of object proposals in the same image associated with one of the feature map layers. The device is configured to obtain proposals, identify a first set of object proposals, including a primary object proposal and one or more other object proposals, based on the indicated object class confidence scores of each of the multiple object proposals, obtain a ranking of feature map layers from the least important to the most important, and determine an attribute class for a first attribute based on the attribute class confidence scores of one or more members of the first set, wherein the determination of the attribute class for the first attribute includes, as part of the determination, a ranking of feature map layers related to one or more other object proposals, and taking into account object location overlaps between the primary object proposal and one or more other object proposals. Therefore, the device is configured to carry out the method of the first embodiment.

[0023] In one or more embodiments of the device, the processing circuit is further configured to cause the device to carry out any embodiment of the method described herein.

[0024] In one or more embodiments of the device, the processing circuit may be further configured to cause the device to implement ANN entities.

[0025] In one or more embodiments of this device, the device is a surveillance camera.

[0026] A third aspect of this disclosure provides a computer program for classifying object attributes in an image. The computer program, when executed by a device's processing circuit (such as the device of the second aspect), causes the device to obtain a plurality of object proposals in the same image from the output of an artificial neural network (ANN) entity trained to locate and classify objects in an image using a plurality of feature map layers associated with different spatial resolutions, wherein each object proposal includes at least i) an object class confidence score for the same first object class, ii) an estimated object location in the image, and iii) an attribute class confidence score for each of one or more different attribute classes for the same first attribute, and each object proposal is associated with one of the feature map layers, thereby obtaining a plurality of object proposals in the same image. The computer program is configured to cause the device to perform the following: identify a first set of object proposals, including a principal object proposal and one or more other object proposals, from among multiple object proposals and based on the indicated object class reliability scores of each of them; obtain a ranking of feature map layers from the least important to the most important; and determine an attribute class for a first attribute based on the attribute class reliability scores of one or more members of the first set, wherein the determination of the attribute class for the first attribute includes, as part of the determination, a ranking of feature map layers related to one or more other object proposals, and taking into account object location overlaps between the principal object proposal and one or more other object proposals. Accordingly, the computer program is configured to cause the device to perform the method of the first embodiment.

[0027] In one or more embodiments of the computer program, the computer program may be further configured, when executed by the device's processing circuit, to cause the device to carry out any embodiment of the first embodiment of the method described herein.

[0028] A fourth aspect of this disclosure provides a computer program product, which includes a computer-readable storage medium for storing a computer program (e.g., computer program code) according to the third aspect (or any embodiment of the third aspect). The computer-readable storage medium used herein is non-temporary and may be provided, for example, as a hard disk drive (HDD), solid-state drive (SSD), USB flash drive, SD card, CD / DVD, and / or any other storage medium capable of non-temporary storage of data. In other embodiments, the computer-readable storage medium may be temporary and correspond to signals (electrical, optical, mechanical, or similar) present, for example, on a communication link, wire, or similar means of signal transmission, in which case the computer-readable storage medium is, of course, a data carrier rather than a data storage entity.

[0029] Other objectives and advantages of this disclosure will become apparent from the embodiments, drawings, and claims for carrying out the invention described below. Within the scope of this disclosure, it is assumed that all features and advantages described, for example, with respect to the methods of the first embodiment are also relevant to, apply to, and be used in combination with, the devices of the second embodiment, the computer programs of the third embodiment, and the computer program products of the fourth embodiment, and vice versa.

[0030] An exemplary embodiment is described below with reference to the attached drawings. [Brief explanation of the drawing]

[0031] [Figure 1]This is a schematic diagram illustrating an exemplary scene image, including two objects that are positioned close together and are of different colors. [Figure 2] Figures A and B schematically illustrate various exemplary ANN architectures (or entities) according to embodiments of the present disclosure. [Figure 3A] This figure schematically illustrates exemplary data showing object proposals output by the ANN architecture according to an embodiment of the present disclosure. [Figure 3B] This figure schematically shows a visual representation of the object locations proposed based on the data in Figure 3A. [Figure 3C] This figure schematically illustrates the incorrect object attribute classification that results from applying conventional post-processing methods to the data in Figure 3A. [Figure 3D] This figure schematically illustrates the improved object attribute classification based on the data in Figure 3A according to the embodiments of this disclosure. [Figure 4] This figure schematically shows a flowchart of an exemplary method for object attribute classification according to an embodiment of the present disclosure. [Figure 5] Figures A and B schematically illustrate exemplary devices for object attribute classification according to embodiments of the present disclosure. [Figure 6] This figure schematically illustrates exemplary computer programs, computer program products, and computer-readable storage media as described herein. [Modes for carrying out the invention]

[0032] In the drawings, similar reference numerals are used for similar elements unless otherwise stated. Unless otherwise explicitly stated, the drawings show only the elements necessary to illustrate exemplary embodiments, and other elements may be omitted or merely suggested for clarity. As shown in the drawings, the (absolute or relative) sizes of elements and areas may be exaggerated or understated relative to their true values ​​for illustrative purposes and are therefore provided to illustrate the general structure of the embodiments.

[0033] In this disclosure, the analysis / processing of a scene image includes at least three operations: i) object positioning, ii) object classification, and iii) classification of one or more object attributes.

[0034] Object positioning involves estimating where an object is positioned within an image of a scene. For example, positioning an object in an image may involve outputting coordinates that define a box surrounding the object (a so-called "bounding box"), or any other way of identifying which pixels in the image are considered to belong to a particular object. For example, if one or more objects are at least partially hidden behind one or more other objects, and similarly, the same set of pixels may, of course, "belong" to multiple objects. A bounding box may be represented, for example, by the coordinates of two of the opposite corners of the bounding box (e.g., x1, y2 for one corner and x2, y2 for the opposite corner), or, for example, by a single coordinate (x, y, etc.) of that box + width (e.g., w) and height (e.g., h) (where the single coordinate is, for example, the center coordinate of that box, or the coordinates of one of its corners), or similar. Those coordinates may be image coordinates. In other examples, the bounding box may instead be represented by an offset from a predefined box, and may include a scaling factor, for example, if the identified bounding box is larger or smaller than the predefined box, and similarly (such as when relying on a so-called anchor box). Other ways of representing the estimated location of an object are, of course, possible. Parameters such as coordinates and / or dimensions (such as height and width) may also be accompanied, in some examples, by their respective uncertainty estimates in coordinates and / or dimensions, as well as similar ones.

[0035] Object classification involves estimating which object class an object is most likely to belong to. Examples of object classes may include, for example, "human," "dog," "cat," "car," and "motorcycle." More general classes may also exist, such as "living organism," "animal," and "vehicle." The exact object classes used depend, for example, on what type of scene you want to analyze, and the ANN architecture may therefore be trained to work with a specific set of object classes. Object classification may result in, for example, a set of confidence scores, each confidence score being for a particular object class and indicating the certainty that the object belongs to that particular object class. A confidence score may be, for example, a decimal number between 0.0 and 1.0, where 0.0 means the object is most likely not to be a member of that object class, 1.0 means the object is most likely to be a member of that object class, and 0.5 means it is uncertain whether the object belongs to that object class or not. Other representations are, of course, possible, such as integers corresponding to percentages from 0 to 100% (where 0.0 is "0%" and 1.0 is "100%"), or similar, instead of decimal numbers. Thus, as assumed herein, object classification may not be binary, but rather produce several confidence scores for several (predefined) object classes.

[0036] Attributes can include things like an object's "color," "shape," "size," and "orientation," and for each such attribute, multiple attribute classes may be provided. For example, for the attribute "color," a set of possible color attribute classes such as "red," "green," and "blue" may be assumed; for the attribute "shape," a set of possible shape attribute classes such as "square," "circular," and "triangle" may be assumed, and so on. Therefore, attribute classification (for a particular attribute) involves estimating which relevant attribute class a particular attribute belongs to. Just as in the case of object classification, the result of such attribute classification may include one or more confidence scores indicating the certainty that a particular attribute belongs to a particular attribute class for that particular attribute.

[0037] Figure 1 schematically shows an exemplary image 100 of a scene, which is used repeatedly herein to illustrate the concepts underlying this disclosure. The scene captured in image 100 includes two specific objects in the form of a first car 110 and a second car 112. The first car 110 is blue, and the second car 112 is red. The locations of the two objects 110 and 112 are as shown in Figure 1. Thus, a properly performed analysis / processing of image 100 should result in each object 110 and 112 being correctly located, correctly classified with respect to object class (i.e., with the highest confidence score for, for example, object class "car"), and correctly classified with respect to color attribute (i.e., with the highest confidence scores for, for example, color attribute classes "blue" and "red", respectively), assuming that color attribute is the only important attribute in this example.

[0038] Next, with reference to Figures 2A and 2B, an exemplary architecture for how object positioning, object classification, and attribute classification should be performed will be described in more detail.

[0039] Figure 2A schematically illustrates an exemplary artificial neural network (ANN) architecture (or entity) 200 envisioned herein. The architecture 200 includes a bottom-up pathway 210 and a top-down pathway 220. The bottom-up pathway 210 includes a number of convolutional stages 212-1 to 212-M, each of which helps to further reduce the spatial resolution of the input image 100 (the input image 100 is provided to the bottom-most stage 212-M, where M is an integer indicating the total number of stages in the bottom-up pathway 210). Each stage 212-1 to 212-M may include a single ANN layer or a set of interconnected ANN layers. The bottom convolutional stage 212-M may reduce the spatial resolution of the input image 100 by half, for example, by using a stride of 2. The output of stage 212-M is provided as input to the next convolutional stage 212-(M-1), which may further reduce the spatial resolution (for example, by using a stride of 2, also by 1 / 2), and so on. Thus, the spatial resolution decreases as we move upward in the bottom-up path 210, but instead, the semantic information provided at each stage increases. The bottom-up path, or at least the upper portion of the bottom-up path, is responsible for performing feature extraction at various levels for different scales of the object (i.e., it is trained to perform that feature extraction), resulting in multiple feature maps at different resolutions. The bottom-up path 210 could be, for example, a residual neural network (i.e., ResNet, which is described in more detail in Kaiming He et al., Deep Residual Learning for Image Recognition, https: / / doi.org / 10.48550 / arXiv.1512.03385), an auto-adjustment network (i.e., RegNet).This could be Jing Xu et al., RegNet: Self-Regulated Network for Image Classification, https: / / doi.org / 10.48550 / arXiv.2101.00590, or any other suitable network architecture for providing feature maps based on different scales / resolutions. The bottom-up path 210 and the various stages 212-1 to 212-M of the bottom-up path 210 are sometimes referred to as the "backbone" of architecture 200, for example.

[0040] The top-down path 220 includes multiple stages 222-1 to 222-N (where N is an integer indicating the total number of such stages, and each stage may include one or more interconnected ANN layers). Each such stage 222-j receives the feature map of the corresponding stage 212-j of the bottom-up path 210 as input, and also receives the output from higher stages in the top-down path 210 (except for the top-most stage 222-1). The top-down path 220 and its stages 222-1 to 222-N may be referred to as the "necks" of the architecture 200, for example, and may provide various upsampling and concatenation mechanisms for fusing feature maps of different stages. In other words, the backbone 210 performs initial feature extraction from the input image 100 at different scales, while the necks 220 perform a higher-performance feature map FM by, for example, performing multi-resolution integration of the features extracted by the backbone 210. jMerge into. Directly providing the outputs from higher-resolution stages 212-(N+1)~212-M of the bottom-up path 210 to the top-down path 220 can often be considered too computationally expensive, so the number of stages M is usually less than N. The top-down path 220 can be implemented, for example, according to a so-called feature pyramid network (FPN, which is described in more detail in Tsung-Yi Lin et al., Feature Pyramid Networks for Object Detection, https: / / doi.org / 10.48550 / arXiv.1612.03144), or an extended variant of the FPN (for example, BiFPN, which is described in more detail in Mingxing Tan et al., EfficientDet: Scalable and Efficient Object Detection, https: / / doi.org / 10.48550 / arXiv.1911.09070).

[0041] Feature Map FM j This is then provided as input to several “head” modules of the architecture 200, including a group 230 of head modules responsible for object classification and localization, and a group 240 of one or more head modules responsible for object attribute classification. Group 230, for example, is used for feature map FM. j The module includes an object classification module 232 configured to determine whether there are objects in image 100 based on the above, and if there are objects in image 100, to attempt to determine which object class those objects belong to. Module 232 includes multiple feature maps FM j It may provide multiple such classifications arising from different feature maps. For example, module 232 provides feature map FM j Based on a specific feature map among them, it provides one or more reliability scores for objects belonging to one or more object classes, and also provides feature map FM. j’≠jBased on certain other feature maps among them, one or more other reliability scores for objects belonging to one or more object classes can be provided and can be configured to operate in the same manner as follows. Such an object classification module 232 can be, for example, a Single Shot Detector (SSD) that uses a convolutional predictor for detection (this is described in more detail in Wei Liu et al., SSD: Single Shot MultiBox Detector, https: / / doi.org / 10.48550 / arXiv.1512.02325), or, for example, Retinanet (described in more detail in Tsung-Yi Lin et al., Focal Loss for Dense Object Detection, https: / / doi.org / 10.48550 / arXiv.1708.02002) or be based thereon. Group 230 can further include an object localization module 234 configured to locate where an object is in image 100 and provide the result, for example, in the form of a bounding box described previously herein. Such an object localization module 234 can, for example, provide a bounding box based on a certain feature map among feature maps FM j and provide another bounding box based on a certain other feature map among feature maps FM j’≠j and can operate in the same manner as follows. Such an object localization module 234 can also be based on either of the two exemplary sources provided above for object classification module 232, but can focus, for example, on predicting an offset from an anchor box rather than object class prediction or something similar.

[0042] Group 240 may include a first attribute classification module 242 configured to determine which attribute class a particular first attribute belongs to, for example, the attribute “color” as previously described herein. Group 240 may also optionally include one or more additional attribute classification modules, such as a second attribute classification module 244 configured to classify certain other attributes. For example, the second attribute classification module 244 may be configured to classify texture attributes, shape attributes, size attributes, attributes relating to whether an object is on a road or not, some other contextual attributes, and this disclosure is not limited to a particular set of attributes and / or corresponding attribute classes. Attribute classification modules 242 and 244 in Group 240 may, for example, feature map FM, just as in the case of modules 232 and 234. j Outputs a set of confidence scores for one or more attribute classes based on a specific feature map among them, and the feature map FM j’≠j It may be configured to output another set of confidence scores for one or more attribute classes based on certain other feature maps among them, and so on. Such attribute classification modules (e.g., 242, 244, etc.) may be based on the same exemplary techniques as those described above for group 230, for example. Another example of how an attribute classification module should be implemented is described in Dong Liu et al., Simultaneous object localization and attribute classification using multitask deep neural networks, US11087130B2.

[0043] As a result of the outputs from the modules of groups 230 and 240, the architecture 200 may be configured to provide data showing multiple object proposals, each object proposal including an estimated object location (from module 234), one or more object class confidence scores (from module 232) for objects belonging to one or more object classes, and one or more attribute class confidence scores (from modules 242 and / or 244) for one or more attribute classes for a particular attribute, in particular each such object proposal includes a feature map FM j It is based on a specific feature map among them.

[0044] Figure 2B schematically illustrates another exemplary ANN architecture (or entity) 201 assumed herein, which is more general than architecture 200 described with reference to Figure 2A. Architecture 201 emphasizes that, as long as there are at least several network layers 214 (or stages) (such as 212-1 to 212-N) operating at different resolutions (e.g., by extracting features), it is not important in this disclosure how the backbone 210 and neck 220 of architecture 200 are structured, or whether, for example, both a backbone and a neck exist. The outputs from such network layers / stages 212-1 to 212-N are then feature-mapped FM to, for example, head modules of groups 230 and 240. j It may be provided directly as, or, for example, as a feature map FM. j Before being output as a feature map FM, it may be further processed (for example, by using the FPN described with reference to Figure 2A, or by any other suitable sub-architecture). Therefore, in general, in this specification, each feature map FM jHowever, as shown in Figure 2B, the exact configuration / layout of each such feature map layer / stage 216-j is not important, as long as the feature map layers / stages 216-j relate to different spatial resolutions, and each object proposal can therefore be referred to as relating to a specific feature map layer / stage among multiple feature map layers / stages 216-j.

[0045] As also shown in Figure 2B, architecture 201 (or any architecture envisioned herein) is envisioned to include, for example, only a single attribute classification module (such as 242).

[0046] Next, with reference to Figures 3A, 3B, 3C, and 3D, exemplary results of analyzing the scene image 100 using architecture 200 or 201, as well as how the methods assumed herein can improve upon modern solutions for post-processing such results, will be described in more detail.

[0047] Figure 3A shows an exemplary output from architecture 200 / 201 in the form of tabular data 300. In data 300, each row corresponds to an object proposal and includes a proposal index ("#", i.e., bounding box index), an estimated object location ("location", in the form of coordinates for two opposite corners of a box, center coordinates + dimensions, etc., as previously described herein but not explicitly shown in Figure 3A), an identifier ("layer") of the feature map layer (or layer / stage 220-1 to 220-N) from which the object proposal originated (i.e., is related), a confidence score for one or more object classes ("object", the confidence score for each object class is provided as "<object class>":<confidence score>), and a confidence score for at least one attribute that is hereby assumed to be a color attribute ("color attribute", the confidence score for each color attribute class is provided as "<color attribute class>":<confidence score>). Here, confidence scores for object classes and color attribute classes are provided using Javascript object notation (JSON), although other formats are also possible. Regarding object classes, only the confidence score for the object class "car" is shown; similarly, such confidence scores may or may not be provided for one or more other object classes.

[0048] Using the first row as an example, we can see from data 300 that this corresponds to a first object proposal (because # is equal to "1") (with a specific estimated location), that this object proposal originated from a first feature map layer (because the layer is equal to "1", i.e., the first object proposal is related to feature map layer 216-1 shown in Figure 2B, for example), that it was given a confidence score of 0.79 for the object class "car" (because the object contains an item with the key "car" and the value "0.79"), and that the confidence scores for the color attribute classes are 0.48 for the attribute class "red" and 0.95 for the attribute class "blue" (because the color attribute contains an item with the key "red" and the value "0.47", as well as another item with the key "blue" and the value "0.95"). Similarly, the second row of data 300 corresponds to a second object proposal generated / related to a second feature map layer (e.g., 216-2), which has a confidence score of 0.88 for the object class "car" and confidence scores of 0.85 and 0.84 for the color attribute classes "red" and "blue," respectively. The third row of data 300 corresponds to a third object proposal generated / related to a third feature map layer (e.g., 216-3), which has a confidence score of 0.82 for the object class "car" and confidence scores of 0.64 and 0.97 for the color attribute classes "red" and "blue," respectively. The fourth row of data 300 corresponds to a fourth object proposal generated from a second feature map layer, which has a confidence score of 0.77 for the object class "car" and confidence scores of 0.98 and 0.32 for the color attribute classes "red" and "blue," respectively. The fifth row of data 300 corresponds to the fifth object proposal generated from the second feature map layer, which has a confidence score of 0.87 for the object class "car" and confidence scores of 0.96 and 0.71 for the color attribute classes "red" and "blue," respectively.The sixth row of data 300 corresponds to a sixth object proposal generated from the third feature map layer, which has a confidence score of 0.85 for the object class "car" and confidence scores of 0.92 and 0.87 for the color attribute classes "red" and "blue," respectively. Data 300 may, of course, contain one or more additional object proposals, but such additional object proposals are ignored in this example. In this particular example, it can therefore be assumed that the architecture (e.g., 201) contains three feature map layers 216-1 to 216-3, i.e., N=3.

[0049] More generally, as assumed in this specification, the architecture used to analyze image 100 includes N feature map layers, and therefore, the corresponding set of different feature map layers F = [F1, F2, ..., F N ] exists (for example, F j It can be further assumed that the feature map layer 216-j in Figure 2B is O=[O1,O2,...,O J ], a set of one or more different attributes A = [A1, A2, ..., A P ] and each such attribute A a Regarding a set of one or more different attribute classes There is TIFF0007867528000002.tif5170, and here, K a Attribute A a It can be assumed that this is the number of possible attribute classes for the data. In that case, data such as data300 each represents the i-th object proposal OP i Multiple lines corresponding to r i This may include each object proposal OP. i This is at least the estimated location L of the object. i , the identifier f of the feature map layer where the object proposal occurred i (For example f i →F) One or more confidence scores OC for each of Oj∈O of one or more object classesi,j , and one or more attributes A a and the corresponding attribute class B a,k One or more reliability scores (AC) for each of them i,a,k This can be shown. For example, using the data 300 shown in Figure 3A, we have F=[F1,F2,F3], O=[O1="car",...], A=[A1="color",...], and B1=[B 1,1 = "Red", B 1,2 = "Blue",...] and OC 1,1 =0.79, OC 2,1 =0.88, OC 3,1 =0.82, OC 4,1 =0.77, OC 5,1 =0.87, OC 6,1 =0.85, ... and AC 1,1,1 =0.47, AC 1,1,2 =0.95, AC 2,1,1 =0.85, AC 2,1,2 =0.84, AC 3,1,1 =0.64, AC 3,1,2 =0.97, AC 4,1,1 =0.98, AC 4,1,2 =0.32, AC 5,1,1 =0.96, AC 5,1,2 =0.71, AC 6,1,1 =0.92, AC 6,1,2 =0.87, ...and so on.

[0050] Figure 3B schematically shows exemplary object locations proposed by object proposals in data 300 as multiple bounding boxes 320. The first object proposal OP1 corresponds to the first bounding box 321, the second object proposal OP2 corresponds to the second bounding box 322, the third object proposal OP3 corresponds to the third bounding box 323, the fourth object proposal OP4 corresponds to the fourth bounding box 324, the fifth object proposal OP5 corresponds to the fifth bounding box 325, the sixth object proposal OP6 corresponds to the sixth bounding box 326, and the label 330 in the upper left corner of each bounding box 320 indicates the corresponding object proposal and row number in data 300. Generally speaking, each object proposal OP i Estimated location L i However, it can be seen here that it is provided as the corresponding bounding box 32i.

[0051] In summary, Figures 3A and 3B indicate that there may be several different bounding boxes (i.e., several different object proposals) for each real-world object 110 and 112, and that post-processing of the data 300 is necessary to identify / estimate which of these many proposed bounding boxes (and object proposals) corresponds to the actual location of that object.

[0052] One conventional technique commonly used for such tasks is called Non-Maximum Suppression (NMS), which relies on computing a so-called Intersection / Union (IOU) measure to identify which of several bounding boxes corresponds to the most likely true location of the object. In short, NMS and IOU work as follows:

[0053] When given a list of object suggestions, such as table data 300, which row corresponds to a specific object class Op The first thing checked is whether it corresponds to the highest confidence score for ∈O. In the current example, a particular object class can be assumed to be a particular object class for "cars," i.e., O1. After examining the data 300, the highest confidence score O i,1 However, it can be found on row i=2 (i.e., OC 2,1 =0.88>OC i≠2,1 Therefore, location L2, which corresponds to bounding box 322, is selected as the principal bounding box for the object belonging to the object class "car". As the next step, the IOU measure is calculated for all other object proposals, which involves finding the overlap between each bounding box 321, 323-326 and the principal bounding box 322. Such overlap can be defined for two bounding boxes as the ratio of the area of ​​the intersection between the two bounding boxes to the combined area of ​​those two boxes. For example, the intersection between two bounding boxes BB1 and BB2 may be defined as |BB1∩BB2|, and the combined area of ​​the two bounding boxes may be defined as |BB1∪BB2|, where the estimated IOU for those two boxes is the IOU 1,2 This results in the equation being |BB1∩BB2| / |BB1∪BB2|. In other words, the object proposal OP does not correspond to the main bounding box 322. i≠2 Such intersection / union IOU for each of them i=2,i≠2 A set of these can be calculated. Of course, the IOU of the main bounding box and itself is equal to 1 by such definition, i.e., IOU 2,2 = 1.0.

[0054] Once IOUs are found between each of the bounding boxes 321, 323-326 and the main bounding box 322, the next step in NMS operation is for each of these IOUs to meet a predefined threshold (e.g., IOU thThis includes checking whether the IOU exceeds the primary bounding box. th If the bounding box exceeds 321, the corresponding bounding box among 323-326 is abandoned from the list and will not be considered further. The IOU with the primary bounding box is the IOU th Bounding boxes 321, 323-326, and their corresponding bounding boxes that do not exceed this limit are also discarded from the list, but instead are placed on a list of potential candidate bounding boxes (i.e., candidate object suggestions) for one or more other objects in the scene.

[0055] As an example, various IOUs are IOUs 2,1 =0.54, IOU 2,3 =0.61, IOU 2,4 =0.02, IOU 2,5 =0.07, and IOU 2,6 The threshold can be estimated as =0.11. th It can be assumed that = 0.2, and therefore only bounding boxes 321 and 323 have a threshold IOU. thIt can be concluded that there is overlap with the main bounding box 322, exceeding [a certain value]. In that case, bounding boxes 321 and 323 are assumed to be inaccurate localizations of the same object as bounding box 322, due to the sufficiently large overlap with the location indicated by object proposal OP2 and bounding box 322. Similarly, the other boxes 324-326 are not considered to be inaccurate localizations of the same object as bounding box 322 and are instead added to the list of candidate proposals for one or more other objects. The above process is then repeated for this other list. Of the bounding boxes 324-326 in data 300 and their corresponding object proposals OP4, OP5, and OP6, object proposal OP5 and bounding box 325 are determined to correspond to the highest confidence score for object class "car" (i.e., OC). 5,1 =0.87). Then, the new IOUs for the remaining boxes 324 and 326 are, for example, IOUs 5,4 =0.43 and IOU 5,6 It can be determined as =0.69, the same threshold IOU th By using =0.2, both of these measures have a predefined threshold IOU th If it exceeds this, it may be further determined, which results in both proposals OP4 and OP6 and their bounding boxes being abandoned from this other list and no longer considered.

[0056] As a result of performing such conventional NMS operations on data 300, object proposal OP2 is therefore selected as the primary proposal for one object, object proposal OP5 is selected as the primary proposal for another object, and the other object proposals are abandoned. For example, for a color attribute, the corresponding attribute class confidence scores found in OP2 and OP5 are examined to determine what the most likely attribute class is, and for object 110 in object proposal OP2, the most likely attribute class for the color attribute is "red" (AC). 2,1,1 =0.85>AC 2,1,2 It can be found that = 0.84. Similarly, for object 112 of object proposal OP5, likewise the most likely attribute class for the color attribute is "red" (AC 5,1,1 =0.96>AC 5,1,2 (Since = 0.71). Note that there is not necessarily a concept of "same" or "different" objects in data 300, and the purpose of the NMS routine is to determine whether there may be two or more objects in the scene image 100.

[0057] Therefore, the results of such a conventional NMS-based analysis are schematically shown in Figure 3C, where the first object 110 is identified as belonging to the object class "car" and its most likely color attribute class is "red," and the second object 112 is also identified as belonging to the object class "car" and its most likely color attribute is also "red," and as shown in Figure 3C, they have locations provided by object proposals OP2 and OP5 and corresponding bounding boxes 322 and 325, respectively. Thus, although both object 110 and object 112 are correctly located and classified with respect to the object class, it can be seen that the color attribute classification of object 110 is incorrect, since it is assumed that the color attribute classification is red and not blue. Therefore, this example helps to show how conventional methods for object location and classification can fail, especially when relying on conventional NMS.

[0058] Next, how this disclosure improves upon modern and conventional methods for object location and classification will be described in more detail.

[0059] As the inventors have noticed, conventional methods (such as those described above with reference to Figure 3C in particular) do not seem to take into account that while certain feature map layers may be "good" at providing useful input for estimating / classifying certain types of attributes, the same feature map layers may be "not so good" at providing useful input for estimating / classifying one or more other types of attributes. For example, higher resolution layers may be better at providing useful outputs for classifying attributes such as color and texture. On the other hand, lower resolution layers may be better at providing useful outputs for classifying "contextual attributes" that contain more semantic information and require more context, i.e., attributes indicating whether an object is on a road or not, which would benefit from, for example, a mixture of information related to both the car and its surrounding elements (such as the road) found in lower resolution layers. Similarly, lower resolution layers may be better at providing more useful outputs for object classification, as it can be difficult to detect objects in higher resolution images that still have too much detail and less semantic content.

[0060] Using Figures 3A and 3B as an example, the best localization of object 110 was provided as part of object proposal OP2 based on the output from a second feature map layer (e.g., 222-2), while the best classification of the color attribute was provided as part of object proposal OP3 based on the output from a third feature map layer (e.g., 222-3). This is consistent with what was stated above, as the third feature map layer 222-3 has a higher spatial resolution than the second feature map layer 222-2, and the second feature map layer provides more semantic information than the third feature map layer. However, by using conventional NMS, the higher quality prediction / classification of the color attribute provided by object proposal OP3 was simply abandoned in favor of the prediction / classification provided by object proposal OP2, as OP2 provides a higher confidence score for classifying the object as belonging to the object class "car". The reason why object proposal OP2 provided a higher confidence score for the color attribute class "red" is that, for example, part of the red object / car 112 is within the bounding box 322, or, for example, the background around the blue object / car 110 is more reddish, or similar, all based on features accumulated in the output from the second feature map layer 222-2, due to a lower resolution input to the second feature map layer 222-2, resulting from additional convolutions performed higher up in the bottom-up path 210 compared to the input provided to the third feature map layer 222-3.

[0061] To overcome or at least partially mitigate the above problems, this disclosure proposes improved post-processing of data such as data 300 to better classify attributes, and also takes into account the source of object proposals (e.g., from which feature map layer). Such improved post-processing will then be described in more detail with reference to Figures 4, 5A, and 5C.

[0062] Figure 4 schematically shows a flowchart of an exemplary method 400 for object attribute classification in an image, such as in image 100. As part of operation S410, method 400 obtains a plurality of object proposals (such as object proposals OP1 to OP6) in the same image from the output of an ANN entity (such as architecture 200) trained to localize an object in the image and classify that object using a plurality of feature map layers (such as layer / stage 316-j). Each i-th object proposal OP i has an object class reliability score (such as OC p ) for the same first object class (e.g., O i,p ∈ O), an estimated object location in the image (e.g., L i ), and an attribute reliability score for each of one or more different attribute classes for the same first attribute (e.g., A q ∈ A), where (e.g., AC i,a,k , where A a ∈ A and B a,k ∈ B a ). Moreover, each object proposal OP i is associated with one of the feature map layers / stages, i.e., f i → F.

[0063] In operation S420, method 400 selects, among the plurality of object proposals OP i , and their respective indicated object class reliability scores CO i,pBased on this, the method includes identifying a first set of object proposals, which include a primary object proposal and one or more other object proposals. For example, using data 300 and Figure 3B as an example, the first set may include object proposal OP2 as the primary object proposal and object proposals OP1 and OP3 as one or more other object proposals. In other examples, one or more of object proposals OP4-OP6 may also be considered part of one or more other object proposals. In other words, method 400 includes identifying the most likely object location (based on the object class confidence score) and identifying one or more other proposals that do not correspond to the most likely object location.

[0064] In operation S430, which may be performed before operation S420, after operation S420, or concurrently with operation S420, method 400 includes ranking feature map layers from the least important to the most important. The ranking may be the same for all attributes, for example, or the ranking may differ for different attributes. For example, if the attribute is "color", the feature map layers may be ranked according to the spatial resolution of the feature map layer, with layers having a higher spatial resolution being ranked as more important than layers having a lower resolution. For example, considering the first to third feature map layers used in the examples in Figures 3A and 3B, it may be assumed that for the color attribute, the first feature map layer is the least important, the third feature map layer is the most important, and the second feature map layer is ranked somewhere between the first and third layers. The ranking may be based on a weighting coefficient w, for example, as will be described in more detail later herein. r This can be done by assigning each r-th feature map layer, or similarly. When ranking feature map layers, in some situations, of course, there may be two or more feature map layers with the same ranking.

[0065] In operation S440, method 400 determines an attribute class for a first attribute based on one or more attribute class reliability scores AC for (all) members of a first set i,a,k (i.e., classifies the first attribute as belonging to a particular attribute class for that attribute). This includes, as part of this determination, ranking feature map layers associated with one or more other object proposals, as well as considering object location overlap (e.g., IOU) between a primary object proposal and one or more other object proposals.

[0066] Rather than simply automatically ranking the feature map layer that is responsible for providing the most likely object location (e.g., the highest object class reliability score) as the most important (or only) feature map layer for classifying an attribute, the present disclosure provides a solution that can improve upon problems with modern solutions illustrated and described, for example, with reference to FIG. 3C. This is by ranking feature map layers according to how "good" they are at providing a useful output for classifying a particular type of attribute.

[0067] As previously explained herein, in some examples of method 400, the location of an object proposal may be provided as a bounding box, and determining object overlap may include determining and using IOU. For example, as already shown, this may include calculating the IOU for each i-th one of one or more other object proposals that are not the i'-th object proposal considered to be the primary object proposal i’,i≠i’ can be included.

[0068] In some examples of Method 400, taking into account ranking and object location overlap may include the fact that attribute class confidence scores indicated by an object proposal, associated with a feature map layer that has greater object overlap with the primary object proposal and is ranked as more important, are more important for determining the attribute class of a first attribute than attribute class confidence scores indicated by an object proposal, associated with a feature map layer that has less overlap with the primary object proposal and is ranked as less important. For example, the primary object proposal is the i'th object proposal OP i’ And two of one or more other object proposals are each the j-th object proposal OP j and the proposed object OP for the j'th object j’ If that is the case, IOU i’,j and IOU i’,j’ Calculate IOU i’,j’ >IOU i’,j It can be determined that this is the case. Furthermore, feature map layer f j’ is layer f j Assuming it was ranked as more important than, attribute class reliability score AC j’,a,k’ Attribute A a When classifying, reliability score A j,a,k It should be given greater importance, and here we can conclude that k' and k may be different or not different, and vice versa.

[0069] As assumed herein, for example, as part of operation S440 of method 400, a specific attribute A a One specific example of how to more accurately determine the correct attribute class for a given can be explained as follows:

[0070] Specific attribute A a One or more attribute classes B about a Each specific attribute class B a,k ∈B aRegarding this, it is across a first set of members that show the attribute class reliability score for that particular attribute class, i.e., attribute class reliability score AC j,a,k It is repeated across the members of the first set having the attribute A. Using the examples in Figures 3A and 3B, a particular attribute is, for example, color attribute A. a=1 This is possible, and therefore, one or more attribute classes are B a=1 =[B 1,1 ,B 1,2 ] and for example, they could be attribute classes "red" and "blue," respectively. If the first set is defined as containing object proposals OP1~OP3, then the iteration is therefore that all of OP1~OP3 have an attribute class reliability score of AC. j,a,k This can be performed over j=1,2,3. For example, if we consider the first set to also include object proposals OP4~OP6, the iteration can instead be performed over j=1,2,...6.

[0071] For each iteration, i.e., for each j, a term T is equal to or proportional to the product of the object location overlap size between the first set of members and the main proposal, the overall ranking score for the feature map layer associated with the member, and the attribute class confidence score for the specific attribute class k indicated by the member. j,a,k This can be determined. For example, term T j teeth, It can be written as TIFF0007867528000003.tif8170, here, TIFF0007867528000004.tif8170 is the overlap between object proposal j and the main object proposal i', where, The file is TIFF0007867528000005.tif8170, where R(f) is a function that assigns ranking values ​​to the feature map layer f, where TIFF0007867528000006.tif8170 is the attribute class confidence score shown by object proposal j for attribute class k for attribute a.

[0072] Next, a specific attribute a and attribute class k (where B k ∈A a Overall attribute class score S for (which is) a,k The overall attribute class score S may be determined. a,k This is the term T determined during the iteration described above across the members of the first set of members that represent the attribute class reliability score for that particular attribute class. j It is equal to or proportional to the sum of S. For example, the overall attribute class score is S a,k =Σ j T j,a,k , or can be determined as similar, where in this or these particular examples, j=1,2,3, or, for example, j=1,2,...6.

[0073] Ultimately, the classification of a specific attribute is then, for example, attribute class B as part of operation S440. a,k’ This can be done by selecting attribute class B a,k’ Regarding the corresponding overall attribute class score S a,k It is the highest, that is, therefore, The filename is TIFF0007867528000007.tif8170.

[0074] In some examples, iterations over j result in an attribute class reliability score of AC. j,a,k However, all other attribute class reliability scores for the same attribute a are AC j,a,k’≠k This can only be done over object proposals that are larger than S. For example, in the tabular data 300 in Figure 3A, this is S 1,1 When calculating the overall score for the color attribute class "red" for the attribute "color", iterate only across object proposals OP2, OP4, OP5 and OP6, S 1,2 When calculating the overall score for the color attribute class "blue" for the same attribute "color", it will involve iterating only across object proposals OP1 and OP3.

[0075] In some examples, the overall attribute class score S for each specific attribute class k a,k S can be defined as being inversely proportional to the number of members in the first set being iterated over. For example, if iterated over J members (i.e., object proposals), then S a,k It can be assumed that ∝ 1 / J.

[0076] In some examples, object proposal OP j Specific feature map layer f related to j Overall ranking score R(f j The rank score R(F) for a particular m-th feature map layer can be defined as the ratio of the ranking score for a feature map layer to the sum of such ranking scores for all feature map layers. For example, the ranking score R(F) for a particular m-th feature map layer m ∈F) is, It can be determined that it is equal to TIFF0007867528000008.tif6170, where w l is the rank score assigned to the l-th feature map layer, and M is the total number of feature map layers.

[0077] In some examples, the overall attribute class score S a,k teeth, TIFF0007867528000009.tif12170 or, for example, It can be defined as TIFF0007867528000010.tif18170, Here, iterations over j result in the corresponding attribute class reliability score AC. j,a,k All object proposals that have j Regarding, or AC j,a,k The attribute class reliability score for all attribute classes k' ≠ k for a specific attribute a is AC. j,a,k’≠k Larger than, Object Proposal OP j It is either only over the period of S. a,kIn the above formula, J is the total number of iterated object proposals. In some examples, object proposals with an IOU of 0 with the principal object proposal are those object proposals with an IOU of 0. i’,j Since these terms would, in any case, negate the contribution of these object proposals to the overall attribute class score, they can be excluded from the iteration.

[0078] In some examples, the ranking of multiple feature map layers may depend on the first attribute a, i.e., R(f j )→R a (f j This can be useful because whether a particular feature map layer is considered to provide a "good" or "not so good" output for classifying a particular attribute may depend on the type of attribute. For example, as mentioned earlier, a higher resolution feature map layer may be better at providing usable output for classifying things like color, texture, and the like, but may perform worse at providing usable output for classifying more contextual attributes, such as whether an object is located on a road or the like, and vice versa. Therefore, the ranking of feature map layers may be modified depending on the specific attribute a of interest.

[0079] In some examples, finding the primary object proposals of the first set and one or more other object proposals can be done using NMS as previously described herein. For example, the particular object class O p The highest object class reliability score for OC j,p An object proposal possessing a certain characteristic may be selected as the primary object proposal, and one or more other object proposals may be discarded from the list if their overlap with the primary object proposal is large enough (i.e., a predefined threshold IOU). thabove), can be defined as an object proposal. In other examples, one or more object proposals can be all other object proposals regardless of whether the overlap of one or more object proposals with the main object proposal is greater than a threshold IOU th or not.

[0080] When reconsidering the examples of FIGS. 3A and 3B as a verification of the proposed method, object proposal OP2 is the main object proposal, and IOU 2,1 = 0.54, IOU 2,3 = 0.61, IOU 2,4 = 0.02, IOU 2,5 = 0.07, and IOU 2,6 = 0.11 (with IOU 2,2 = 1.0 by definition) can be determined. To classify the attribute "color" (i.e., A1) for the bounding box 322 corresponding to the main object proposal OP2, for example, the above equation (1) can be used. The exemplary ranking of feature map layers 1, 2, and 3 can be w1 < w2 < w3 because higher resolution layers are considered better at outputting useful data for color attribute classification. For example, if we assume w3 = 0.6, w2 = 0.3, and w1 = 0.1, obtain that it is TIFF0007867528000011.tif10170 and TIFF0007867528000012.tif10170. Therefore, since the overall score S 1,2 for the color attribute class "blue" is greater than the overall score S 1,1 for the color attribute class "red", object 110 will then be correctly classified as blue rather than red, contrary to the result obtained using only conventional NMS. Instead, when iterating over all object proposals, not only those object proposals whose overlap exceeds IOU th but also all object proposals, obtain that it is TIFF0007867528000013.tif10170 and It is obtained that the filename is TIFF0007867528000014.tif10170. In other words, object 110 is S 1,2 >S 1,1 Therefore, it will still be correctly classified as "blue." For confirmation, after selecting object proposal OP5 as the primary object proposal for object 112, (IOU 5,1 =0.02, IOU 5,2 =0.07, IOU 5,3 =0.03, IOU 5,4 =0.43, and IOU 5,6 = 0.69, and according to the definition, IOU 5,5 (After determining that = 1.0), TIFF0007867528000015.tif10170 and It is TIFF0007867528000016.tif10170. or TIFF0007867528000017.tif10170 and It can also be confirmed that the filename is TIFF0007867528000018.tif10170. (Of course, including OP5) Is the iteration performed across all object proposals, or is the IOU the IOU? th Regardless of whether it extends beyond object proposals OP4 and OP6, object 112 is S 1,1 >S 1,2 Therefore, we confirm that it will be correctly classified as red.

[0081] As assumed herein, the ranking of feature map layers may be performed manually, for example, based on user experience, or in a more automated manner, for example. l It is assumed that feature map layer scores (such as those mentioned above) can be obtained by using an exhaustive search method to find optimized weights / scores. For example, score w lAssuming that it should be between 0 and 1, the automated procedure initiated by this assumption will have a score of w l =1 to make all such scores equal. Then, for lower resolution layers, the above procedure may proceed by exploring from, for example, 0.9 to 1.0 with intervals of 0.05 to find the score corresponding to the best mean precision (AP) value or similar for the test dataset. As an example, hyperparameter optimization may be used to explore a particular set of parameters (e.g., feature map layer weights / scores) to produce optimal performance. Such optimization may be carried out using one or more frameworks available for such purposes, such as Optuna (described in more detail in Takuya Akiba et al., Optuna: A Next-generation Hyperparameter Optimization Framework, https: / / doi.org / 10.48550 / arXiv.1907.10902).

[0082] This specification also envisions providing devices, computer programs, and computer program products for object attribute classification in images, which will be described in more detail with reference to Figures 5A, 5B, and 6.

[0083] Figure 5A schematically shows a device 500 for object attribute classification in an image, as previously described herein, i.e., a device configured to perform, for example, method 400 as described with reference to Figure 4. Device 500 includes at least a processor (or “processing circuit”) 510 and optionally a memory 512. As used herein, “processor” or “processing circuit” may be one or any combination of a suitable central processing unit (CPU), multiprocessor, microcontroller (μC), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), graphics processing unit (GPU), etc., capable of executing software instructions stored in memory 512. Memory 512 may be outside or inside the processor 510. As used herein, “memory” may be random access memory (RAM) and read-only memory (ROM), or any combination of any other type of memory capable of storing instructions. Memory 512, when executed by processor 510, contains (i.e., stores) instructions causing device 500 to carry out the method described herein (i.e., method 400 or any embodiment of method 400). Device 500 may further include one or more additional items 514 that may be useful in carrying out the method in some circumstances. In some exemplary embodiments, device 500 may be a (video) camera, such as a (video) surveillance camera, and the (one or more) additional items 514 may in that case include, for example, an image sensor and, for example, one or more lenses for focusing light from the scene onto the image sensor, so that the surveillance camera may capture an image of the scene as part of carrying out the assumed method.The (one or more) additional items 514 may also include various other electronic components required to capture the scene, for example, to properly operate the image sensor and / or lens as needed, and to collect information relating to the current orientation and / or focal length of the camera and (one or more) lenses, which may be used, for example, to determine the camera's current FOV. Implementing the method in a surveillance camera may be useful in that the processing is moved to the “edge,” i.e., closer to where the actual scene is captured, compared to performing post-processing of object proposals elsewhere (such as in a more centralized processing server or similar). Device 500 may be connected to a network, for example, so that the results from implementing the method can be transmitted to a user. For this purpose, device 500 may include a network interface 516, which may be, for example, a wireless network interface (as defined in either IEEE 802.11 or a subsequent standard, for example, supporting Wi-Fi) or a wired network interface (as defined in either IEEE 802.3 or a subsequent standard, for example, supporting Ethernet). The network interface 516 may also support any other wireless standards capable of transmitting encoded video, such as Bluetooth or similar. The various components 510, 512, 514, and 516 (if any) may be connected via one or more communication buses 520, so that these components can communicate with each other and exchange data as needed.

[0084] Device 500 could be a building-mounted or mountable surveillance camera, for example, in the form of a PTZ camera, or a fisheye camera capable of providing a wider view of a scene, or any other type of surveillance camera. Device 500 could be a body camera, action camera, dashcam, or similar, suitable for mounting on people, animals and / or various vehicles, or similar. Device 500 could be a smartphone or tablet, for example, that a user can carry and use to capture scenes. In any such example of Device 500, it is assumed that Device 500 may include all other necessary components (if any) not already described herein, insofar as Device 500 is still capable of carrying out Method 400 or any embodiment of Method 400 as envisioned herein. Various components of Device 500 may be further configured in some examples to implement the ANN architecture / entity described herein, for example 200 or 201. In other examples, device 500 might simply be configured to receive output from such an ANN architecture / entity and perform post-processing of multiple object proposals.

[0085] Figure 5B schematically shows one or more embodiments of device 500 relating to several function / calculation blocks 510a to 510d. Each such block 510a to 510d is responsible for performing a function by a specific operation of method 400, as shown in the flowchart of Figure 4. For example, one such function block 410a may be configured to retrieve output from an ANN entity (operation S410), another block 510b may be configured to identify members of a first set (operation S420), another block 510c may be configured to rank (or retrieve the ranking of) a feature map layer (operation S430), and another block 510d may be configured to determine an attribute class for a first attribute (operation S440). Device 500 may optionally include one or more additional function blocks 510e, such as blocks for implementing an ANN architecture or similar.

[0086] Generally speaking, each functional module 510a to e can be implemented in hardware or software. Preferably, one or more or all of the functional modules 510a to e can be implemented by the processing circuit 510, optionally in cooperation with a storage medium / memory 512 and / or a communication interface 516. The processing circuit 510 is therefore configured to fetch instructions provided by the functional modules 510a to e from the memory 512 and execute these instructions, thereby enabling it to perform any operation of method 400 as performed by / in device 500 as disclosed herein.

[0087] Figure 6 schematically shows a computer program product 610 including a computer-readable means / storage medium 630. A computer program 620 may be stored on the computer storage medium 630, and the computer program 620 can cause the processor 510, and entities and devices operably coupled to the processor 510, such as the communication interface 516 and memory 512 of the device 500, to perform the method 400 according to embodiments described with reference to, for example, Figures 2A, 2B, 3A, 3B, 3D, and 4. Thus, the computer program 620 and / or computer program product 610 may provide means for performing any operation of the method 400 performed by the device 500 as disclosed herein.

[0088] In the example in Figure 6, the computer program product 610 and the computer-readable storage medium 630 are shown as optical discs, such as CDs (Compact Discs), DVDs (Digital Multipurpose Discs), or Blu-ray discs. The computer program product 610 and the computer-readable storage medium 630 can also be embodied as memory, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM), and more specifically as non-volatile storage media in external memory devices, such as USB (Universal Serial Bus) memory or flash memory such as CompactFlash memory. Thus, although the computer program 620 is schematically shown here as a track on the shown optical disc, the computer program 620 can be stored in any way that is suitable for the computer program product 610 and the computer-readable storage medium 630.

[0089] To summarize the various embodiments presented herein, this disclosure provides an improved way of post-processing object proposals from ANN architectures / entities that utilize feature maps and feature map layers for multiple spatial resolutions. In particular, this disclosure proposes not to simply discard information in object proposals just because a particular feature map (layer) used to predict an object class is not particularly good at that task, because the same feature map (layer) may simultaneously be excellent at accurately classifying one or more attributes of an object. This is because the task of object detection (often based on identifying contours and shapes instead of things, such as color, texture, and similar things) is often more preferably performed on lower-resolution images, while the task of attribute classification (such as identifying color, texture, and similar things) may be more preferably performed on higher-resolution images where such information (regarding color, texture, etc.) is not yet lost. By ranking the feature map layers (in terms of their ability to perform attribute classification) and taking into account how well each object proposal overlaps with the main object proposal (in terms of object location), the risk of misclassifying a particular attribute as belonging to the wrong attribute class can be reduced. Another advantage is that the proposed solution does not necessarily require modifying existing ANN-based architectures already used to provide object proposals, but can instead be implemented only as post-processing for such object proposals.

[0090] While features and elements may be described above in specific combinations, each feature or element may be used alone without other features and elements, or in various combinations with or without other features and elements. In addition, variations of the disclosed embodiments may be understood and realized by those skilled in the art in carrying out the claimed invention, based on the drawings, this disclosure, and the accompanying study of the claims.

[0091] In the claims, the words “comprising” and “including” do not exclude other elements, and the indefinite article “a” or “an” does not exclude the plural. The mere fact that several features are described in different dependent claims does not imply that combinations of these features cannot be used to one's advantage. [Explanation of symbols]

[0092] 100 Scene Images 110, 112 objects 200, 201 ANN architecture 210 Bottom-up routes 212-j Folding section 214 Multiple network layers 216-j Feature map layer / stage 220 Top-down route 222-j Folding section 230 Head Group 232 Object Classification Module 234 Object Location Module 240 Head Group 242, 244 Attribute Classification Modules 300 data 320-326 bounding boxes 330 Bounding Box Index 400 ways S410, 420, 430, 440 operation 500 devices 510 Processing Circuit 512 memory 514 Additional circuits / functions 516 Interface 510a~e Functional Blocks 610 Computer Program Products 620 Computer Programs 630 Storage medium

Claims

1. A method (400) for classifying object attributes in an image (100) performed by a computer, - Obtaining multiple object proposals in the same image from the output of artificial neural network (ANN) entities (200, 201) trained to locate and classify objects (110, 112) in an image using multiple feature map layers (216-j) associated with different spatial resolutions (S410), wherein each object proposal represents at least i) an object class confidence score for the same first object class, ii) an estimated object location in the image, and iii) an attribute class confidence score for each of one or more different attribute classes for the same first attribute, and each object proposal is associated with one of the feature map layers (S410), - Identifying a first set of object proposals, including a principal object proposal and one or more other object proposals, based on the multiple object proposals and the respective object class reliability scores they represent (S420), - Regarding the ability to extract information useful for classifying the first attribute, the feature map layers are ranked from the least important feature map layer to the most important feature map layer (S430), - Determining an attribute class for the first attribute based on one or more attribute class reliability scores of all members of the first set (S440), which includes, as part of the determination, taking into account the ranking of the feature map layer related to the one or more other object proposals, and the object location overlap between the main object proposal and the one or more other object proposals (S440) Includes, Determining the attribute class for the first attribute is - For each specific attribute class of the one or more attribute classes mentioned above, The attribute class reliability score for the specific attribute class is shown, and this is repeated across the members of the first set. For each iteration, determine a term that is equal to or proportional to the product of the object location overlap size between the member of the first set and the main object proposal, the overall ranking score for the feature map layer associated with the member, and the attribute class confidence score for the specific attribute class indicated by the member, and The overall attribute class score for the aforementioned specific attribute class is determined to be equal to or proportional to the sum of the terms determined during the iteration, - The attribute class for the first attribute is determined to be the attribute class having the highest overall attribute class score. Method (400), further including the following:

2. The method according to claim 1, wherein the object locations of each object proposal are shown as bounding boxes (320-326), and the method further comprises using an inter-intersection / union (IOU) operation to determine the overlap of the object locations between the principal object proposal (322) and one or more other object proposals (321, 323-326).

3. The method of claim 1 or 2, wherein taking into account the ranking and object location overlaps, the attribute class confidence score indicated by the object proposal, relating to a feature map layer having greater object overlap with the primary object proposal and ranked as more important, is more important for determining the attribute class of the first attribute than the attribute class confidence score indicated by the object proposal, relating to a feature map layer having less object overlap with the primary object proposal and ranked as less important.

4. The method according to claim 1 or 2, wherein the iteration is performed only over the first set of members that show the highest attribute class confidence score for the particular attribute class.

5. The method according to claim 1 or 2, wherein the overall attribute class score for each specific attribute class is further defined as being inversely proportional to the number of members in the first set that are repeated.

6. The method according to claim 1 or 2, wherein the overall ranking score for the feature map layer associated with the member is defined as the ratio of the ranking score for the feature map layer to the sum of such ranking scores for all of the plurality of feature map layers.

7. The method according to claim 1 or 2, wherein the ranking of the plurality of feature map layers depends on the first attribute.

8. The method according to claim 1 or 2, further comprising using non-maximal suppression (NMS) operation to identify the primary object proposal and the one or more other object proposals.

9. The method according to claim 1 or 2, wherein the ANN entity comprises a feature pyramid network (FPN) (220) for providing the plurality of feature map layers.

10. The method according to claim 1 or 2, wherein the ANN entity comprises a plurality of convolutional layers (210) for providing the different spatial resolutions of the image.

11. A device (500) for classifying object attributes in an image (100), wherein the device comprises a processing circuit (510), and the processing circuit (510) provides the device, - Obtaining multiple object proposals in the same image from the output of artificial neural network (ANN) entities (200, 201) trained to locate objects (110, 112) in an image (100) using multiple feature map layers (216-j) associated with different spatial resolutions, wherein each object proposal represents at least i) an object class confidence score for the same first object class, ii) an estimated object location in the image, and iii) an attribute class confidence score for each of one or more different attribute classes for the same first attribute, and each object proposal is associated with one of the feature map layers, - Identifying a first set of object proposals, including a primary object proposal (322) and one or more other object proposals (321, 323-326), based on the respective object class reliability scores of the multiple object proposals, - Regarding the ability to extract information useful for classifying the first attribute, this involves obtaining a ranking of the feature map layers from the least important feature map layer to the most important feature map layer, - Determining an attribute class for a first attribute based on one or more attribute class confidence scores of all members of the first set, comprising, as part of the determination, taking into account the ranking of the feature map layer relating to the one or more other object proposals, and object location overlap between the principal object proposal and the one or more other object proposals. It is configured to perform the following: Determining the attribute class for the first attribute is - For each specific attribute class of the one or more attribute classes mentioned above, The attribute class reliability score for the specific attribute class is shown, and this is repeated across the members of the first set. For each iteration, determine a term that is equal to or proportional to the product of the object location overlap size between the member of the first set and the main object proposal, the overall ranking score for the feature map layer associated with the member, and the attribute class confidence score for the specific attribute class indicated by the member, and The overall attribute class score for the aforementioned specific attribute class is determined to be equal to or proportional to the sum of the terms determined during the iteration, - The attribute class for the first attribute is determined to be the attribute class having the highest overall attribute class score. A device (500) further includes the following.

12. The device according to claim 11, wherein the processing circuit is further configured to cause the device to implement the ANN entities (200, 201).

13. A computer program (620) for classifying object attributes in an image (100), wherein the computer program, when executed by a processing circuit (510) of a device (500), provides the device, - Obtaining multiple object proposals in the same image from the output of artificial neural network (ANN) entities (200, 201) trained to locate and classify objects in an image (100) using multiple feature map layers (216-j) associated with different spatial resolutions, wherein each object proposal represents at least i) an object class confidence score for the same first object class, ii) an estimated object location in the image, and iii) an attribute class confidence score for each of one or more different attribute classes for the same first attribute, and each object proposal is associated with one of the feature map layers, - Identifying a first set of object proposals, including a primary object proposal (322) and one or more other object proposals (321, 323-326), based on the respective object class reliability scores of the multiple object proposals, - Regarding the ability to extract information useful for classifying the first attribute, this involves obtaining a ranking of the feature map layers from the least important feature map layer to the most important feature map layer, - Determining an attribute class for a first attribute based on one or more attribute class confidence scores of all members of the first set, comprising, as part of the determination, taking into account the ranking of the feature map layer relating to the one or more other object proposals, and object location overlap between the principal object proposal and the one or more other object proposals. It is configured to perform the following: Determining the attribute class for the first attribute is - For each specific attribute class of the one or more attribute classes mentioned above, The attribute class reliability score for the specific attribute class is shown, and this is repeated across the members of the first set. For each iteration, determine a term that is equal to or proportional to the product of the object location overlap size between the member of the first set and the main object proposal, the overall ranking score for the feature map layer associated with the member, and the attribute class confidence score for the specific attribute class indicated by the member, and The overall attribute class score for the aforementioned specific attribute class is determined to be equal to or proportional to the sum of the terms determined during the iteration, - The attribute class for the first attribute is determined to be the attribute class having the highest overall attribute class score. Computer programs (620) further include.

14. A computer-readable storage medium (630) in which the computer program (620) described in claim 13 is stored.

Images