Improved object attribute classification

Abstract

To provide a method for classifying object attributes in an image.SOLUTION: A method includes: obtaining a plurality of object proposals from an artificial neural network entity trained to localize and classify objects using a plurality of feature map layers associated with different spatial resolutions (S410); identifying a main object proposal and one or more other object proposals (S420); ranking the feature map layers from a least significant feature map layer to a most significant feature map layer (S430); and determining an attribute class for a first attribute based on attribute class confidence scores of the main and other proposals (S440). The determination of the attribute class for the first attribute includes taking the ranking of the feature map layers as well as object location overlaps into account.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure generally relates to the field of image processing. In particular, the present disclosure relates to the localization and classification of objects in images, as well as to the classification of object attributes.

Background Art

[0002] Artificial neural network (ANN)-based architectures have been found to be useful for detecting instances of (semantic) objects of several object classes in images, such as still images or image frames of videos. Examples of object classes include, for example, humans, animals, vehicles, license plates, faces, and the like. In addition to merely determining whether an object of a certain object class may be present in an image, such architectures can also be trained to estimate where in the image the object is located. An example of such a modern architecture is the Feature Pyramid Network (FPN), which is considered to be particularly suitable for detecting objects at different scales.

[0003] ANN-based architectures can also be used to classify one or more attributes, features, and even activities of an object. Object attributes can include, for example, color, texture, object shape, and can include, for example, whether the object is currently facing a certain direction, whether the object is moving or stationary, whether the object is positioned on a particular type of surface (such as a road, sidewalk, etc.), or any other details of the object or its location that can be categorized and labeled, such as context. Attribute classification is considered an important task, for example, in computer vision, because, for example, appropriate classification of attributes can help distinguish different objects present in the same scene.

[0004] However, modern ANN-based architectures often cannot distinguish between different objects and instead work by providing a number of object proposals. Each such proposal may include, for example, the proposed object location, a confidence score for one or more object classes, and a confidence score for one or more attribute classes for one or more attributes. Thus, post-processing of the output from the ANN-based architecture is required to determine, for example, how many distinct objects are in an image, where those objects are positioned, and the most likely object and / or attribute classes for that object. Such known post-processing operations include the use of so-called non-maximum suppression (NMS) and intersection-over-union (IOU) measures.

[0005] This disclosure aims to develop such post-processing of object proposals from ANN-based architectures and mitigate one or more shortcomings of modern techniques.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Non-Patent Documents

[0007]

Non-Patent Document 1

[0008] For the purposes described above, the present disclosure proposes an improved method, device, computer program, and computer program product for object attribute classification in images, as defined by the appended independent claims. Various embodiments are defined by the appended dependent claims.

[0009] According to a first aspect of the present disclosure, a method for object attribute classification in an image is provided. The method uses a plurality of feature map layers associated with different spatial resolutions to locate an object in the image and obtains a plurality of object proposals in the same image from the output of an artificial neural network (ANN) entity trained to classify the object, wherein each object proposal indicates at least: i) an object class reliability score for the same first object class; ii) an estimated object location in the image; and iii) an attribute class reliability score for each of one or more different attribute classes for the same first attribute, and includes obtaining a plurality of object proposals in the same image, each object proposal being associated with one of the feature map layers. The method further includes identifying a first set including a primary object proposal and one or more other object proposals among the plurality of object proposals and based on their respective indicated object class reliability scores. The method further includes ranking the feature map layers from the least significant feature map layer to the most significant feature map layer. The method further includes determining an attribute class for the first attribute based on the one or more attribute class reliability scores of all members of the first set, and as part of the determining, ranking the feature map layers associated with the one or more other object proposals and taking into account object location overlaps between the primary object proposal and the one or more other object proposals, and further includes determining an attribute class for the first attribute.

[0010] As will be described in more detail later in this specification, the proposed method improves upon currently available techniques in that it does not simply ignore information related to object attributes found in one or more object proposals that are not considered to be primary object proposals. Instead, the proposed method uses this information together with the information provided by the primary object proposal, and in particular, when determining the final estimated attribute class for a particular first attribute of an object, also considers whether information generated from a particular feature map layer should be given more weight or less weight. As will be illustrated later in this specification, this reduces the risk of misclassifying attributes, for example, when two or more objects are present in the same part of an image.

[0011] In one or more embodiments of the method, the object location of each object proposal can be represented as a bounding box, and the method can further include using an intersection over union (IOU) measure / operation to determine object location overlap between the primary object proposal and one or more other object proposals. Such measures are generally known, readily available, and can be efficiently implemented on modern hardware.

[0012] In one or more embodiments of the present method, taking into account ranking and object location duplication may include that the attribute class reliability score indicated by an object proposal, which has a greater duplication with a major object proposal and is ranked as more important and is related to a feature map layer, is made more important than the attribute class reliability score indicated by an object proposal, which has a smaller object duplication with the major object proposal and is ranked as less important and is related to a feature map layer, for determining the attribute class of the first attribute. In other words, the supposed method includes that this object proposal should have a sufficiently large duplication (regarding the supposed object location) with the major object proposal and should originate from a feature map layer ranked as more relevant (i.e., better in determining the first attribute of the object) for the information of a specific object proposal to be regarded as more relevant for determining the attribute class for the first attribute. This is compared with any other object proposal that originated from a feature map layer ranked as having a smaller duplication with the major object proposal and being less relevant (i.e., worse in determining the first attribute of the object) for determining the first attribute of the object. For example, if a specific feature map layer is regarded as better (i.e., operating at a resolution more suitable for determining the color of the object) in determining, for example, the color of the object, the information from the object proposal originating from this specific feature map layer is given more weight in determining the attribute class for the first attribute as long as these object proposals also propose locations that are sufficiently similar to the locations proposed by the major object proposal.

[0013] In one or more embodiments of the present method, determining an attribute class for a first attribute may further include iterating over members of a first set that indicate an attribute class reliability score for a particular attribute class, for each particular attribute class of the one or more attribute classes. For each iteration, determining a term equal to or proportional to the product of the object location overlap size between a member of the first set and the primary object proposal, the overall ranking score for the feature map layer associated with the member, and the attribute class reliability score for the particular attribute class indicated by the member, and determining the overall attribute class score for the particular attribute class as equal to or proportional to the sum of the terms determined during the iteration. The method may further include determining the attribute class for the first attribute as the attribute class having the highest overall attribute class score. Such a procedure may help implement the above concept of assigning higher importance to object proposals originating from a feature map layer that closely overlaps with the primary object proposal (with respect to object location) and is considered to be better and more relevant for classifying the first attribute.

[0014] In one or more embodiments of the present method, the iteration may be performed only over members of the first set that indicate the highest attribute class reliability score for a particular attribute class. In other words, it is envisioned that the iteration is performed only over members of the first set for which a particular attribute class has the top 1 attribute class reliability score. In other examples, it may instead be iterated over all members of the first set, regardless of whether a particular attribute class has the top 1 attribute class reliability score.

[0015] In one or more embodiments of the present method, the overall attribute class score for each specific attribute class may further be defined as being inversely proportional to the number of members of the repeated first set. For example, if it is repeated over a total of P members, the overall attribute class score may be scaled by a factor of 1 / P, or the like.

[0016] In one or more embodiments of the present method, the overall ranking score for a feature map layer associated with a member may be defined as the ratio of the ranking score for the feature map layer to the sum of such ranking scores for all of the plurality of feature map layers. For example, if each l-th feature map layer is assigned a ranking score w l then the overall ranking score for the l-th feature map layer may be defined as TIFF2025106196000002.tif5170.

[0017] In one or more embodiments of the present method, the ranking of the plurality of feature map layers may depend on a first attribute. In other words, the ranking of the feature map layers may vary for different attributes, which helps to take into account that different feature map layers may be more or less relevant (i.e., better or worse) for the classification of a particular type of attribute.

[0018] In one or more embodiments of the present method, the method may further include using a non-maximum suppression (NMS) operation to identify a primary object proposal and one or more other object proposals.

[0019] In one or more embodiments of the present method, the ANN entity may include a feature pyramid network (FPN) for providing a plurality of feature map layers. As described above, the FPN may be particularly useful for detecting objects at different (spatial) scales, such as both larger and smaller objects in the same image.

[0020] In one or more embodiments of the method, the ANN entity comprises a plurality of convolutional layers for providing different spatial resolutions.

[0021] In one or more embodiments of the method, the method is implemented in / by a surveillance camera.

[0022] According to a second aspect of the present disclosure, a device for object attribute classification in an image is provided. The device includes a processing circuit (such as a processor), and the processing circuit causes the device to use a plurality of feature map layers associated with different spatial resolutions to locate an object in the image and classify the object from the output of an artificial neural network (ANN) entity trained to do so (e.g., as a result of executing instructions stored in the device's memory), obtaining a plurality of object proposals in the same image, where each object proposal indicates at least: i) an object class reliability score for the same first object class; ii) an estimated object location in the image; and iii) an attribute class reliability score for each of one or more different attribute classes for the same first attribute, obtaining a plurality of object proposals in the same image associated with one of the feature map layers, identifying a first set including a primary object proposal and one or more other object proposals among the plurality of object proposals and based on their respective indicated object class reliability scores, obtaining a ranking of the feature map layers from the least important feature map layer to the most important feature map layer, determining an attribute class for the first attribute based on one or more attribute class reliability scores of all members of the first set, including as part of the determining, taking into account the ranking of the feature map layers associated with one or more other object proposals and the object location overlap between the primary object proposal and the one or more other object proposals, and causing the device to perform the determining. Thus, the device is configured to implement the method of the first aspect.

[0023] In one or more embodiments of the device, the processing circuit is further configured to cause the device to implement any embodiment of the methods described herein.

[0024] In one or more embodiments of the present device, the processing circuit may be further configured to cause the present device to implement an ANN entity.

[0025] In one or more embodiments of the present device, the present device is a surveillance camera.

[0026] According to a third aspect of the present disclosure, there is provided a computer program for object attribute classification in an image. When executed by a processing circuit of a device (such as the device of the second aspect), the computer program causes the device to locate an object in an image using a plurality of feature map layers associated with different spatial resolutions, and obtain a plurality of object proposals in the same image from the output of an artificial neural network (ANN) entity trained to classify the object, where each object proposal indicates at least: i) an object class reliability score for the same first object class; ii) an estimated object location in the image; and iii) an attribute class reliability score for each of one or more different attribute classes for the same first attribute, obtaining a plurality of object proposals in the same image associated with one of the feature map layers, identifying a first set including a primary object proposal and one or more other object proposals among the plurality of object proposals and based on their respective indicated object class reliability scores, obtaining a ranking of the feature map layers from the least important feature map layer to the most important feature map layer, determining an attribute class for the first attribute based on the one or more attribute class reliability scores of all members of the first set, including, as part of the determining, ranking the feature map layers associated with the one or more other object proposals and taking into account object location overlaps between the primary object proposal and the one or more other object proposals, and causing the device to determine an attribute class for the first attribute. Thus, the computer program is configured to cause the device to implement the method of the first aspect.

[0027] In one or more embodiments of the present computer program, when executed by a processing circuit of a device, the computer program may be further configured to cause the device to implement any embodiment of the method of the first aspect described herein.

[0028] According to a fourth aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer-readable storage medium storing a computer program (e.g., computer program code) according to the third aspect (or any embodiment of the third aspect). The computer-readable storage medium used herein may be, for example, non-transitory and may be provided as, for example, a hard disk drive (HDD), a solid state drive (SSD), a USB flash drive, an SD card, a CD / DVD, and / or any other storage medium capable of non-transitory storage of data. In other embodiments, the computer-readable storage medium may be transitory and may correspond to a signal (electrical, optical, mechanical, or the like) existing, for example, on a communication link, a wire, or a similar means of signal transfer, in which case the computer-readable storage medium is, of course, more of a data carrier rather than a data storage entity.

[0029] Other objects and advantages of the present disclosure will become apparent from the following detailed description of the invention, the drawings, and the claims. It is assumed that within the scope of the present disclosure, all features and advantages described, for example, with respect to the method of the first aspect are equally relevant to, applicable to, and usable in combination with the device of the second aspect, the computer program of the third aspect, and the computer program product of the fourth aspect, and vice versa.

[0030] Exemplary embodiments are described below with reference to the accompanying drawings.

Brief Description of the Drawings

[0031]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 3C

Figure 3D

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0032] In the drawings, like reference numerals are used for like elements unless otherwise noted. Unless explicitly stated otherwise, the drawings show only the elements necessary to illustrate exemplary embodiments, and other elements may be omitted or merely suggested for clarity. As shown in the figures, the (absolute or relative) sizes of elements and regions may be exaggerated or presented conservatively relative to their true values for illustrative purposes, and thus are provided to illustrate the general structure of the embodiments.

[0033] In the present disclosure, the analysis / processing of an image of a scene includes at least three operations, namely, i) object positioning, ii) object classification, and iii) classification of one or more object attributes.

[0034] Object positioning involves estimating where an object is positioned within an image of a scene. As an example, positioning an object in an image can include outputting the coordinates that define a box (so-called "bounding box") that encloses the object, or any other way of identifying which pixels of the image are considered to belong to a particular object. For example, if there are one or more objects that are at least partially hidden behind one or more other objects, and in similar cases, the same set of pixels can, of course, "belong" to multiple objects. The bounding box can be represented, for example, by the coordinates of two of the corners on opposite sides of the bounding box (e.g., x1,y2 for one corner and x2,y2 for the opposite corner, etc.), or, for example, a single coordinate (e.g., x,y) of the box + width (e.g., w) and height (e.g., h) (the single coordinate can be, for example, the center coordinate of the box, or the coordinate of a corner of the box), or can be represented in a similar way. Those coordinates can be image coordinates. In other examples, the bounding box can instead be represented by an offset from a pre-defined box and can include a scaling factor, for example, if the identified bounding box is larger or smaller than the pre-defined box, and in similar cases (such as when relying on a so-called anchor box). Other ways of representing the estimated location of an object are, of course, possible. Parameters such as coordinates and / or dimensions (such as height and width) can also, in some examples, have an associated uncertainty estimate for each of the coordinates and / or dimensions, and the like.

[0035] Object classification involves estimating which object class an object is most likely to belong to. Examples of object classes can include, for example, "human", "dog", "cat", "car", "motorcycle", etc. There can also be more general classes such as "living thing", "animal", "vehicle", etc. Exactly which object classes are used depends, for example, on what type of scene one wishes to analyze, and the ANN architecture can thus be trained to work with a particular set of object classes. Object classification can result in, for example, a set of confidence scores being provided, each confidence score being for a particular object class and indicating the certainty that the object belongs to that particular object class. The confidence scores can be, for example, decimal numbers between 0.0 and 1.0, where 0.0 means that the object is most likely not a member of that object class, 1.0 means that the object is most likely a member of that object class, and 0.5 means that it is uncertain whether the object belongs to that object class. Other representations are of course possible, such as, instead of decimal numbers, integers corresponding to percentages 0 - 100% (where 0.0 is "0%" and 1.0 is "100%"), or the like. Thus, as envisioned herein, object classification is not binary, but rather can result in several confidence scores for several (pre - defined) object classes.

[0036] Attributes can include things such as the "color", "shape", "size", "orientation" of an object, etc., and for each such attribute, multiple attribute classes can be provided. For example, for the attribute "color", a set of possible color attribute classes such as "red", "green", "blue", etc. can be assumed, and for the attribute "shape", a set of possible shape attribute classes such as "square", "circle", "triangle", etc. can be assumed, and so on for the following. Therefore, attribute classification (for a particular attribute) involves inferring to which related attribute class a particular attribute belongs. Just like in the case of object classification, the result of such attribute classification can include one or more confidence scores indicating the certainty that a particular attribute belongs to a particular attribute class for that particular attribute.

[0037] FIG. 1 schematically shows an exemplary image 100 of a scene that is repeatedly used herein to illustrate the concepts underlying the present disclosure. The scene captured in image 100 includes two particular objects in the form of a first vehicle 110 and a second vehicle 112. The first vehicle 110 is blue and the second vehicle 112 is red. The locations of the two objects 110 and 112 are as shown in FIG. 1. Thus, a correctly performed analysis / processing of image 100 should result in each object 110 and 112 being correctly positioned, correctly classified with respect to the object class (i.e., with the highest confidence score for the object class "vehicle", for example), and, assuming that the color attribute is the only important attribute in this example, correctly classified with respect to the color attribute (i.e., with the highest confidence score for the color attribute classes "blue" and "red", respectively).

[0038] Next, with continued reference to FIGS. 2A and 2B, an exemplary architecture for how object positioning, object classification, and attribute classification should be performed will be described in more detail.

[0039] Figure 2A schematically shows an exemplary artificial neural network (ANN) architecture (or entity) 200 contemplated herein. Architecture 200 includes a bottom-up pathway 210 as well as a top-down pathway 220. The bottom-up pathway 210 includes a plurality of convolutional stages 212-1 to 212-M, each of which serves to further reduce the spatial resolution of the input image 100 (the input image 100 is provided to the bottom-most stage 212-M, where M is an integer indicating the total number of stages in the bottom-up pathway 210). Each stage 212-1 to 212-M may include a single ANN layer or a set of interconnected ANN layers). The bottom-most convolutional stage 212-M may reduce the spatial resolution of the input image 100, for example, by a factor of 1 / 2 by using a stride of 2, for example. The output of stage 212-M is provided as an input to the next convolutional stage 212-(M-1), which may further reduce the spatial resolution (by, for example, also a factor of 1 / 2 by using a stride of 2, for example), and so on. Thus, the spatial resolution decreases as it moves upward in the bottom-up pathway 210, but instead, the semantic information provided at each stage increases. The bottom-up pathway, or at least the upper portion of the bottom-up pathway, is responsible for performing feature extraction at various levels for different scales of the object (i.e., is trained to perform that feature extraction) and generates a plurality of feature maps at different resolutions. The bottom-up pathway 210 may be, for example, a residual neural network (i.e., ResNet, which is described in more detail in Kaiming He et al., Deep Residual Learning for Image Recognition, https: / / doi.org / 10.48550 / arXiv.1512.03385), an adaptive network (i.e., RegNet.This is more detailedly described in Jing Xu et al., RegNet: Self-Regulated Network for Image Classification, https: / / doi.org / 10.48550 / arXiv.2101.00590, or can be any other suitable network architecture for providing feature maps based on different scales / resolutions. The bottom-up path 210 and the various stages 212-1 to 212-M of the bottom-up path 210 may be referred to, for example, as the "backbone" of the architecture 200.

[0040] The top-down path 220 includes a plurality of stages 222-1 to 222-N (where N is an integer indicating the total number of such stages, and each stage may include one or more interconnected ANN layers). Each such stage 222-j receives as input the feature map of the associated stage 212-j of the bottom-up path 210, and also receives as input the output from a higher stage in the top-down path 210 (except for the top-most stage 222-1). The top-down path 220 and the stages 222-1 to 222-N of the top-down path 220 may be referred to, for example, as the "neck" of the architecture 200 and may provide various upsampling and concatenation mechanisms for fusing feature maps of different stages. In other words, the backbone 210 performs initial feature extraction from the input image 100 at different scales, while the neck 220 performs, for example, multi-resolution integration of those features by performing multi-resolution integration of the features extracted by the backbone 210 to obtain a higher-performance feature map FM. jMerge into it. Directly providing the output from the higher-resolution stages 212-(N+1) to 212-M of the bottom-up path 210 to the top-down path 220 can often be regarded as too computationally expensive, so the number of stages M is usually smaller than N. The top-down path 220 can be implemented according to, for example, the so-called Feature Pyramid Network (FPN, which is more detailedly described in Tsung-Yi Lin et al., Feature Pyramid Networks for Object Detection, https: / / doi.org / 10.48550 / arXiv.1612.03144), or an extended variant of FPN (for example, BiFPN, which is more detailedly described in Mingxing Tan et al., EfficientDet: Scalable and Efficient Object Detection, https: / / doi.org / 10.48550 / arXiv.1911.09070).

[0041] Feature map FM j is then provided as input to a plurality of "head" modules of the architecture 200, including a group 230 of head modules responsible for object classification and localization, and a group 240 of one or more head modules responsible for object attribute classification. The group 230 includes, for example, an object classification module 232 configured to determine whether there is an object in the image 100 based on the feature map FM j and also to attempt to determine to which object class the object belongs if there is an object in the image 100. The module 232 can provide a plurality of such classifications generated from different feature maps among the plurality of feature maps FM j . For example, the module 232 provides one or more confidence scores for objects belonging to one or more object classes based on a specific feature map among the feature maps FM j , and also the feature map FM j’≠jBased on certain other feature maps among them, one or more other reliability scores for objects belonging to one or more object classes can be provided and can be configured to operate in the same manner as follows. Such an object classification module 232 can be, for example, a Single Shot Detector (SSD) that uses a convolutional predictor for detection (this is described in more detail in Wei Liu et al., SSD: Single Shot MultiBox Detector, https: / / doi.org / 10.48550 / arXiv.1512.02325), or, for example, Retinanet (described in more detail in Tsung-Yi Lin et al., Focal Loss for Dense Object Detection, https: / / doi.org / 10.48550 / arXiv.1708.02002), or can be based on it. Group 230 can further include an object localization module 234 configured to locate where the object is in the image 100 and provide the result, for example, in the form of a bounding box described previously herein. Such an object localization module 234 can, for example, provide a bounding box based on a certain feature map among the feature maps FM j and provide another bounding box based on a certain other feature map among the feature maps FM j’≠j and can operate in the same manner as follows. Such an object localization module 234 can also be based on, for example, either of the two exemplary sources provided above for the object classification module 232, but can focus on, for example, predicting the offset from the anchor box rather than object class prediction, or something similar.

[0042] Group 240 may include a first attribute classification module 242 configured to determine to which attribute class a particular first attribute belongs, such as, for example, the attribute "color" described previously herein. Group 240 may optionally also include one or more additional attribute classification modules, such as a second attribute classification module 244 configured to classify a particular other attribute. For example, the second attribute classification module 244 may be configured to classify texture attributes, shape attributes, size attributes, attributes related to whether an object is on a road, some other context attribute, etc., and the present disclosure is not limited to a particular set of attributes and / or corresponding attribute classes. Just as in the case of modules 232 and 234, the attribute classification modules 242 and 244 in group 240 output a set of reliability scores for one or more attribute classes based on a particular feature map, for example, in the feature map FM j and output another set of reliability scores for one or more attribute classes based on a particular other feature map in the feature map FM j’≠j and may be configured to do so in a similar manner hereinafter. Such attribute classification modules (e.g., 242, 244, etc.) may be based on, for example, the same exemplary techniques described above for group 230. Another example of how to implement the attribute classification module is described in Dong Liu et al., Simultaneous object localization and attribute classification using multitask deep neural networks, US11087130B2.

[0043] As a result of the outputs from the modules of groups 230 and 240, architecture 200 can be configured to provide data indicative of a plurality of object proposals, each object proposal including an estimated object location (from module 234), one or more object class reliability scores (from module 232) for an object belonging to one or more object classes, and one or more attribute class reliability scores (from modules 242 and / or 244) for one or more attribute classes for a particular attribute. In particular, each such object proposal is based on a particular feature map of the feature maps FM j among them.

[0044] FIG. 2B schematically shows another exemplary ANN architecture (or entity) 201 contemplated herein that is more general than architecture 200 described with reference to FIG. 2A. It is emphasized that within the present disclosure, it is not important, for example, how exactly the backbone 210 and neck 220 of architecture 200 are structured, or for example whether both a backbone and a neck are present, as long as there are at least a plurality of network layers 214 (or stages), such as 212-1 to 212-N, that operate at different resolutions (e.g., by extracting features). The outputs from such network layers / stages 212-1 to 212-N are then provided directly as feature maps FM j to, for example, the head modules of groups 230 and 240, or can be further processed (e.g., by using the FPN described with reference to FIG. 2A or by any other suitable sub-architecture) before being output as feature maps FM j . Thus, generally herein, each feature map FM jAs shown in Figure 2B, it is related to a specific feature map layer / stage 216-j, and the exact configuration / layout of each such feature map layer / stage 216-j is not considered important as the feature map layer / stage 216-j is related to different spatial resolutions and, thus, each object proposal can be referred to as being related to a specific feature map layer / stage among the plurality of feature map layers / stages 216-j.

[0045] Also as shown in Figure 2B, it is assumed that the architecture 201 (or any architecture contemplated herein) may include only a single attribute classification module (such as 242).

[0046] Next, with continued reference to Figures 3A, 3B, 3C, and 3D, exemplary results of analyzing an image 100 of a scene using the architecture 200 or 201, and how the methods contemplated herein can improve modern solutions for post-processing such results are described in more detail.

[0047] Figure 3A shows an exemplary output from architecture 200 / 201 in the form of tabular data 300. In data 300, each row corresponds to an object proposal and includes a proposal index (the "#", i.e., the bounding box index), an estimated object location (a "location" in the form of coordinates for two opposite corners of the box, center coordinates + dimensions, etc., which was previously described herein but not explicitly shown in FIG. 3A), an identifier of the feature map layer (or stage of layers / stages 220-1 to 220-N) where the object proposal occurred (i.e., the "layer"), a confidence score for one or more object classes (the "object", with the confidence score for each object class provided as "<object class>":<confidence score>), and a confidence score for at least a first attribute that is here assumed to be a color attribute (the "color attribute", with the confidence score for each color attribute class provided as "<color attribute class>":<confidence score>). Here, the confidence scores for object classes and color attribute classes are provided using JavaScript Object Notation (JSON), although other formats are of course possible. For object classes, only the confidence score for the object class "car" is shown, although it is also possible that such confidence scores for one or more other object classes are provided or not provided.

[0048] Using the first row as an example, from data 300, this corresponds to a first object proposal (with a specific estimated location), and since the # is equal to "1", and that object proposal originated from the first feature map layer (since the layer is equal to "1", i.e., the first object proposal is related to the feature map layer 216-1 shown in, for example, Figure 2B), and a reliability score of 0.79 for the object class "car" is shown (since the object includes an item with key "car" and value "0.79"), and the reliability scores for the color attribute classes are 0.48 for the attribute class "red" and 0.95 for the attribute class "blue" (since the color attribute includes an item with key "red" and value "0.47" and another item with key "blue" and value "0.95"). Similarly, the second row of data 300 corresponds to a second object proposal that originated from / is related to the second feature map layer (such as 216-2) and has a reliability score of 0.88 for the object class "car" and reliability scores of 0.85 and 0.84 for the color attribute classes "red" and "blue" respectively. The third row of data 300 corresponds to a third object proposal that originated from / is related to the third feature map layer (such as 216-3) and has a reliability score of 0.82 for the object class "car" and reliability scores of 0.64 and 0.97 for the color attribute classes "red" and "blue" respectively. The fourth row of data 300 corresponds to a fourth object proposal that originated from the second feature map layer and has a reliability score of 0.77 for the object class "car" and reliability scores of 0.98 and 0.32 for the color attribute classes "red" and "blue" respectively. The fifth row of data 300 corresponds to a fifth object proposal that originated from the second feature map layer and has a reliability score of 0.87 for the object class "car" and reliability scores of 0.96 and 0.71 for the color attribute classes "red" and "blue" respectively.The sixth row of data 300 corresponds to the sixth object proposal generated from the third feature map layer, having a reliability score of 0.85 for the object class "car" and reliability scores of 0.92 and 0.87 for the color attribute classes "red" and "blue", respectively. Data 300 may, of course, include one or more additional object proposals, but the one or more additional object proposals in that case are ignored in this example. In this particular example, therefore, the architecture (e.g., 201) can be assumed to include three feature map layers 216-1 to 216-3, i.e., N = 3.

[0049] More generally, as contemplated herein, the architecture used to analyze image 100 includes N feature map layers, and thus there is a corresponding set F = [F1, F2,..., F N of different feature map layers (where, for example, F j is the feature map layer 216-j in FIG. 2B) can be further assumed. There is a set O = [O1, O2,..., O J of one or more different object classes, a set A = [A1, A2,..., A P of one or more different attributes, and for each such attribute A a there is a set of one or more different attribute classes for TIFF2025106196000003.tif5170, where K a is assumed to be the number of possible attribute classes for attribute A a . In that case, data such as data 300 can include multiple rows r i each corresponding to a particular i-th object proposal OP i . Each object proposal OP i includes at least the estimated location L i of the object, the identifier f i of the feature map layer from which the object proposal was generated (e.g., f i → F), and one or more reliability scores OC for each of one or more object classes Oj ∈ Oi,j and one or more attributes A a and corresponding attribute classes B a,k one or more reliability scores AC for each of i,a,k which may be shown. For example, as an example, using the data 300 shown in FIG. 3A, F = [F1, F2, F3], O = [O1 = "car",...], A = [A1 = "color",...], and B1 = 1,1 = "red", B 1,2 = "blue",...], and OC 1,1 = 0.79, OC 2,1 = 0.88, OC 3,1 = 0.82, OC 4,1 = 0.77, OC 5,1 = 0.87, OC 6,1 = 0.85,... and AC 1,1,1 = 0.47, AC 1,1,2 = 0.95, AC 2,1,1 = 0.85, AC 2,1,2 = 0.84, AC 3,1,1 = 0.64, AC 3,1,2 = 0.97, AC 4,1,1 = 0.98, AC 4,1,2 = 0.32, AC 5,1,1 = 0.96, AC 5,1,2 = 0.71, AC 6,1,1 = 0.92, AC 6,1,2 = 0.87,... are involved.

[0050] FIG. 3B schematically shows exemplary object locations proposed by object proposals of data 300 as a plurality of bounding boxes 320. The first object proposal OP1 corresponds to the first bounding box 321, the second object proposal OP2 corresponds to the second bounding box 322, the third object proposal OP3 corresponds to the third bounding box 323, the fourth object proposal OP4 corresponds to the fourth bounding box 324, the fifth object proposal OP5 corresponds to the fifth bounding box 325, the sixth object proposal OP6 corresponds to the sixth bounding box 326, and the label 330 at the upper left corner of each bounding box 320 indicates the corresponding object proposal and line number in data 300. Generally, the estimated location L of each object proposal OP i can here be seen to be provided as the corresponding bounding box 32i. i FIG. 3A and FIG. 3B are summarized, it should be noted that there can be several different bounding boxes (i.e., several different object proposals) for each real object 110 and 112, and it is necessary to identify / estimate which of the many proposed bounding boxes (and object proposals) correspond to the actual location of the object by post-processing the data 300.

[0051] One conventional technique commonly used for such tasks is called non-maximum suppression (NMS), and NMS relies on calculating a so-called intersection over union (IOU) measure to identify which of a plurality of bounding boxes corresponds to the most likely true location of the object. Briefly summarized, NMS and IOU work as follows.

[0052] When a list of object proposals is provided, such as tabular data 300, which row is a particular object class O

[0053] ​p It is first checked whether it corresponds to the highest confidence score for ∈O. In the current example, a particular object class can be assumed to be a particular object class for "car", i.e., O1. After examining the data 300, the highest confidence score O i,1 is found to be found on row i = 2 (i.e., OC 2,1 = 0.88 > OC i≠2,1 ). Therefore, the location L2 corresponding to the bounding box 322 is selected as the main bounding box for the object belonging to the object class "car". As a next step, the IOU measure for all other object proposals is calculated, which involves finding the overlap between each bounding box 321, 323 - 326 and the main bounding box 322. Such an overlap can be defined as the ratio of the area of the common part between two bounding boxes to the combined area of those two boxes for the two bounding boxes. For example, the common part between two bounding boxes BB1 and BB2 can be defined as |BB1∩BB2|, and the combined area of the two bounding boxes can be defined as |BB1∪BB2|, which results in the estimated IOU for those two boxes being IOU 1,2 = |BB1∩BB2| / |BB1∪BB2|. In other words, a set of such common part / sum set IOU i≠2 for each of the object proposals OP i=2,i≠2 not corresponding to the main bounding box 322 can be calculated. Of course, the IOU between the main bounding box and itself is equal to 1 by such a definition, i.e., IOU 2,2 = 1.0.

[0054] Once the IOU between each of the bounding boxes 321, 323 - 326 and the main bounding box 322 is found, the next step of the NMS operation is that each of these IOUs is compared with a predefined threshold (e.g., IOU thincluding checking whether it exceeds th Among the bounding boxes 321, 323 - 326, the corresponding bounding box whose IOU with the main bounding box exceeds th is discarded from the list and is no longer considered. Among the bounding boxes 321, 323 - 326, the corresponding bounding box whose IOU with the main bounding box does not exceed

[0055] is also discarded from the list, but instead is placed on the list of potential candidate bounding boxes (i.e., candidate object proposals) for one or more other objects in the scene. 2,1 = 0.54, IOU 2,3 = 0.61, IOU 2,4 = 0.02, IOU 2,5 = 0.07, and IOU 2,6 = 0.11 can be estimated as an example. It can be assumed that the threshold is, for example, IOU th = 0.2, and thus only the bounding boxes 321 and 323 exceed the threshold IOU thcan be concluded to have an overlap with the main bounding box 322 that exceeds it. In that case, bounding boxes 321 and 323 are assumed to be an inaccurate localization of the same object as the object of bounding box 322, as bounding boxes 321 and 323 have a sufficiently large overlap with the location indicated by object proposal OP2 and bounding box 322. Similarly, the other boxes 324 - 326 are considered not to be an inaccurate localization of the same object as the object of bounding box 322 and are instead added to the list of candidate proposals for one or more other objects. The process is then repeated for this other list. Of the bounding boxes 324 - 326 in data 300 and the corresponding object proposals OP4, OP5, and OP6 for bounding boxes 324 - 326, it is determined that object proposal OP5 and bounding box 325 correspond to the highest reliability score for object class "car" (i.e., OC 5,1 = 0.87). New IOU values for the remaining boxes 324 and 326 can then be determined, for example, as IOU 5,4 = 0.43 and IOU 5,6 = 0.69, and by using the same threshold IOU th = 0.2, it can be further determined that both of these measures exceed a predefined threshold IOU th , which results in both proposal OP4 and OP6 and the bounding boxes of proposal OP4 and OP6 being discarded from this other list and no longer considered.

[0056] As a result of performing such a conventional NMS operation on the data 300, the object proposal OP2 is thus selected as the main proposal for a certain object, the object proposal OP5 is selected as the main proposal for another object, and the other object proposals are discarded. For example, for the color attribute, in order to determine what the most likely attribute class is, the corresponding attribute class reliability scores found in OP2 and OP5 are examined, and for the object 110 of the object proposal OP2, the most likely attribute class for the color attribute is found to be "red" (AC 2,1,1 =0.85>AC 2,1,2 =0.84). Similarly, for the object 112 of the object proposal OP5, again, the most likely attribute class for the color attribute is "red" (AC 5,1,1 =0.96>AC 5,1,2 =0.71). It should be noted that there is not necessarily a concept of "same" or "different" objects in the data 300, and the purpose of the NMS routine is to determine whether there may be two or more objects in the image 100 of the scene.

[0057] Therefore, the results of such conventional NMS-based analysis are as schematically shown in FIG. 3C, where the first object 110 is identified as belonging to the object class "car" and the most likely color attribute class being "red", while the second object 112 is also identified as belonging to the object class "car" and the most likely color attribute also being "red", and as shown in FIG. 3C, each has a location provided by object proposals OP2 and OP5 and corresponding bounding boxes 322 and 325, respectively. Thus, both object 110 and object 112 are correctly positioned and classified with respect to the object class, but since the color attribute classification of object 110 is assumed to be red rather than blue, it can be seen that the color attribute classification is incorrect. Therefore, this example serves to illustrate how conventional methods for object localization and classification can fail, especially when relying on conventional NMS.

[0058] Next, how the present disclosure improves modern and conventional methods for object localization and classification will be described in more detail.

[0059] As the inventors have realized, conventional methods (such as those described above with particular reference to FIG. 3C) may be "good" in providing useful input for a particular feature map layer to infer / classify a particular type of attribute, but do not appear to take into account that the same feature map layer may be "not so good" in providing useful input for inferring / classifying one or more other types of attributes. For example, higher resolution layers may be better in providing useful output for the classification of attributes such as color and texture. On the other hand, lower resolution layers contain more semantic information and are better in providing useful output for the classification of attributes that require more context, i.e., "contextual attributes", such as an attribute indicating whether an object is on a road, which will benefit from the mixture of information related to both the vehicle and the surrounding elements (such as the road) found in lower resolution layers. Similarly, lower resolution layers may be better in providing more useful output for object classification as it may be difficult to detect objects in higher resolution images that still have too much detail and less semantic content remaining.

[0060] Using FIGS. 3A and 3B as an example, the best localization of object 110 was provided as part of object proposal OP2 based on the output from the second feature map layer (e.g., 222-2), while the best classification of the color attribute was provided as part of object proposal OP3 based on the output from the third feature map layer (e.g., 222-3). This is in line with what was described above because the third feature map layer 222-3 has a higher spatial resolution than the second feature map layer 222-2, and also because the second feature map layer provides more semantic information than the third feature map layer. However, by using conventional NMS, the higher-quality prediction / classification of the color attribute provided by object proposal OP3 was simply discarded in favor of the prediction / classification provided by object proposal OP2 because OP2 provided a higher confidence score for the classification of the object as belonging to the object class "car". The reason why object proposal OP2 provided a higher confidence score for the color attribute class "red" is, for example, because part of the red object / car 112 is within the bounding box 322, or, for example, because the background around the blue object / car 110 is more reddish, or the like, and may be due to, for example, features accumulated in the output from the second feature map layer 222-2 based on the lower-resolution input to the second feature map layer 222-2 resulting from additional convolutions performed higher up in the bottom-up path 210 compared to the input provided to the third feature map layer 222-3.

[0061] To overcome or at least partially mitigate the above problems, the present disclosure proposes improved post-processing of data, such as data 300, to better classify attributes, and also takes into account the source of the object proposals (e.g., from which feature map layer). Such improved post-processing will then be described in more detail with reference to FIGS. 4, 5A, and 5C as well.

[0062] Figure 4 schematically shows a flowchart of an exemplary method 400 for object attribute classification in an image, such as in image 100. As part of operation S410, method 400 obtains a plurality of object proposals (such as object proposals OP1 to OP6 in the same image) from the output of an ANN entity (such as architecture 200) trained to localize an object in the image and classify that object using a plurality of feature map layers (such as layer / stage 316-j). Each i-th object proposal OP i has an object class reliability score (such as OC p for the same first object class (e.g., O i,p ∈ O), an estimated object location (such as L i ) in the image, and an attribute reliability score for each of one or more different attribute classes for the same first attribute (e.g., A q ∈ A) (such as AC i,a,k , where A a ∈ A and B a,k ∈ B a ). Moreover, each object proposal OP i is associated with one of the feature map layers / stages, i.e., f i → F.

[0063] In operation S420, method 400 selects among the plurality of object proposals OP i and their respective indicated object class reliability scores CO i,pincluding identifying a first set that includes a primary object proposal and one or more other object proposals. For example, using data 300 and FIG. 3B as an example, the first set includes object proposal OP2 as the primary object proposal and may include, for example, object proposals OP1 and OP3 as one or more other object proposals. In other examples, one or more of object proposals OP4-OP6 may also be considered part of one or more other object proposals. In other words, method 400 includes identifying the most likely object location (based on the object class reliability score) and also identifying one or more other proposals that do not correspond to the most likely object location.

[0064] Prior to operation S420, after operation S420, or in operation S430 that may be performed simultaneously with operation S420, method 400 includes ranking the feature map layers from the least important feature map layer to the most important feature map layer. The ranking may be the same for all attributes, for example, or the ranking may be different for different attributes. For example, if the attribute is "color", the feature map layers may be ranked according to the spatial resolution of the feature map layer, and a layer with a higher spatial resolution is ranked as more important than a layer with a lower resolution. For example, considering the first to third feature map layers used in the examples of FIGS. 3A and 3B, for the color attribute, the first feature map layer is the least important layer, the third feature map layer is the most important layer, and the second feature map layer may be assumed to be ranked somewhere in between the first and third layers. The ranking may be performed, for example, by assigning a weight factor w r to each r-th feature map layer, or by something similar. When ranking the feature map layers, in some situations, of course, there may be two or more feature map layers with the same ranking.

[0065] In operation S440, method 400 determines an attribute class for a first attribute based on one or more attribute class reliability scores AC for (all) members of a first set i,a,k (i.e., classifies the first attribute as belonging to a particular attribute class for that attribute). This includes, as part of this determination, ranking feature map layers associated with one or more other object proposals, as well as taking into account object location overlap (e.g., IOU) between a primary object proposal and one or more other object proposals.

[0066] Rather than simply automatically ranking the feature map layer that is responsible for providing the most likely object location (e.g., the highest object class reliability score) as the most important (or only) feature map layer for classifying an attribute, the present disclosure provides a solution that can improve on the problems associated with modern solutions exemplified and described with reference to, for example, FIG. 3C. This is by ranking feature map layers according to how "good" they are at providing a useful output for classifying a particular type of attribute.

[0067] As previously explained herein, in some examples of method 400, the location of an object proposal may be provided as a bounding box, and determining object overlap may include determining and using IOU. For example, as already shown, this may include calculating the IOU for each i-th one of one or more other object proposals that are not the i'-th object proposal considered to be the primary object proposal i’,i≠i’ can include.

[0068] In some examples of method 400, taking into account ranking and object location duplication, the attribute class reliability score indicated by an object proposal that has a greater object duplication with a major object proposal and is associated with a feature map layer ranked as more important may be more important than the attribute class reliability score indicated by an object proposal that has a smaller duplication with the major object proposal and is associated with a feature map layer ranked as less important for determining the attribute class of the first attribute. For example, if the major object proposal is the i'-th object proposal OP i’ and two of one or more other object proposals are, respectively, the j-th object proposal OP j and the j'-th object proposal OP j’ then IOU i’,j and IOU i’,j’ can be calculated, and it can be determined that IOU i’,j’ >IOU i’,j . Further, assuming that the feature map layer f j’ is ranked as more important than the layer f j the attribute class reliability score AC j’,a,k’ should be made more important than the reliability score A a when classifying the attribute A j,a,k where k' and k may or may not be different, and it can be concluded that the reverse is also true.

[0069] As assumed herein, for example, as part of operation S440 of method 400, one specific example of how to more accurately determine the correct attribute class for a particular attribute A a can be described as follows.

[0070] For each specific attribute class B a of one or more attribute classes B a for a particular attribute A a,k ∈B aFor it, it is over the members of the first set indicating the attribute class reliability score for that specific attribute class, i.e., the attribute class reliability score AC j,a,k is repeated over the members of the first set having. Using the examples of FIGS. 3A and 3B, a specific attribute can be, for example, color attribute A a=1 and thus one or more attribute classes can be B a=1 =[B 1,1 ,B 1,2 , and can be, for example, the attribute classes "red" and "blue" respectively. If the first set is defined as including object proposals OP1 to OP3, the iteration can thus be performed for j = 1, 2, 3 since all of OP1 to OP3 indicate the attribute class reliability score AC j,a,k . For example, if the first set is considered to also include object proposals OP4 to OP6, the iteration can instead be performed for j = 1, 2,... 6.

[0071] For each iteration, i.e., for each j, a term T equal to or proportional to the product of the object location overlap size between the member of the first set and the main proposal, the overall ranking score for the feature map layer associated with the member, and the attribute class reliability score for the specific attribute class k indicated by the member is determined. j,a,k For example, the term T j can be written as TIFF2025106196000004.tif8170, where TIFF2025106196000005.tif8170 is the overlap between object proposal j and main object proposal i', where TIFF2025106196000006.tif8170, where R(f) is a function that assigns a ranking value to feature map layer f, where TIFF2025106196000007.tif8170 is the attribute class reliability score indicated by object proposal j for attribute class k for attribute a.

[0072] Next, for a particular attribute a and attribute class k (where B k ∈A a ), an overall attribute class score S a,k can be determined, and the overall attribute class score S a,k is equal to or proportional to the sum of terms T j determined during the above-described iteration over members of a first set that indicate the attribute class reliability score for that particular attribute class. For example, the overall attribute class score is S a,k = Σ j T j,a,k , or can be determined similarly, where, in this or these particular examples, j = 1, 2, 3, or, for example, j = 1, 2,... 6.

[0073] Finally, the classification of the particular attribute can then be performed, for example, as part of operation S440, by selecting the attribute class B a,k’ , for which the corresponding overall attribute class score S a,k’ is the highest, i.e., thus, a,k TIFF2025106196000008.tif8170. TIFF2025106196000008.tif8170.

[0074] In some examples, the iteration over j can be performed only over object proposals for which the attribute class reliability score AC j,a,k is greater than all other attribute class reliability scores AC j,a,k’≠k for the same attribute a. For example, as an example, in the tabular data 300 of FIG. 3A, this would include iterating only over object proposals OP2, OP4, OP5, and OP6 when calculating S 1,1 (i.e., the overall score for the color attribute class "red" for the attribute "color") and iterating only over object proposals OP1 and OP3 when calculating S 1,2 (i.e., the overall score for the color attribute class "blue" for the same attribute "color").

[0075] In some examples, the overall attribute class score S for each specific attribute class k a,k may be defined as being inversely proportional to the number of members of the repeated first set. For example, when iterating over J members (i.e., object proposals), S a,k may be assumed to be ∝ 1 / J.

[0076] In some examples, the overall ranking score R(f j for a specific feature map layer f related to an object proposal OP j ) may be defined as the ratio of the ranking score for the feature map layer to the sum of such ranking scores for all of the plurality of feature map layers. For example, the ranking score R(F j ∈ F) for a specific m-th feature map layer is m determined to be equal to TIFF2025106196000009.tif6170, where w l is the ranking score assigned to the l-th feature map layer and M is the total number of feature map layers.

[0077] In some examples, the overall attribute class score S a,k is defined as TIFF2025106196000010.tif12170 or, for example,[ TIFF2025106196000011.tif18170, where the iteration over j is either over all object proposals OP j,a,k having a corresponding attribute class reliability score AC j or only over object proposals OP j,a,k where AC j,a,k’≠k is greater than all attribute class reliability scores AC j for other attribute classes k' ≠ k for a particular attribute a. S a,kIn the above formula for, J is the total number of object proposals to be iterated. In some examples, object proposals with an IOU of 0 with the main object proposal, their 0 IOU i’,j terms will, in any case, cancel out their contribution from these object proposals to the overall attribute class score and can thus be excluded from the iteration.

[0078] In some examples, the ranking of multiple feature map layers may depend on the first attribute a, i.e., thus, R(f j ) → R a (f j ). This can be useful because whether a particular feature map layer is considered to provide a "good" output or a "not so good" output for classifying a particular attribute may depend on the type of that attribute. For example, as described above, a higher resolution feature map layer may be better at providing usable outputs for the classification of, for example, color, texture, and the like, but may function worse at providing usable outputs for classifying more contextual attributes such as whether an object is positioned on a road, and vice versa. Thus, the ranking of the feature map layers can be changed depending on the particular attribute a of interest.

[0079] In some examples, finding the first set of main object proposals and one or more other object proposals can be performed using NMS, as previously described herein. For example, an object proposal with the highest object class reliability score OC p for a particular object class O j,p can be selected as the main object proposal, and one or more other object proposals are such that the overlap with the main object proposal is large enough to be discarded from the list (i.e., a predefined threshold IOU thabove), and can be defined as an object proposal. In other examples, one or more object proposals can be all other object proposals, regardless of whether the overlap of one or more object proposals with the main object proposal is greater than a threshold IOU th or not.

[0080] When reconsidering the examples of FIGS. 3A and 3B as a verification of the proposed method, the object proposal OP2 is the main object proposal, and IOU 2,1 = 0.54, IOU 2,3 = 0.61, IOU 2,4 = 0.02, IOU 2,5 = 0.07, and IOU 2,6 = 0.11 (with IOU 2,2 = 1.0 by definition) can be determined. To classify the attribute "color" (i.e., A1) for the bounding box 322 corresponding to the main object proposal OP2, for example, the above equation (1) can be used. The exemplary ranking of the feature map layers 1, 2, and 3 can be w1 < w2 < w3 because higher resolution layers are considered better in outputting useful data for color attribute classification. For example, assuming w3 = 0.6, w2 = 0.3, and w1 = 0.1, TIFF2025106196000012.tif10170 and TIFF2025106196000013.tif10170 are obtained. Therefore, since the overall score S 1,2 for the color attribute class "blue" is greater than the overall score S 1,1 for the color attribute class "red", the object 110 will then be correctly classified as blue rather than red, contrary to the result obtained using only conventional NMS. Instead, when iterating over all object proposals, not only object proposals whose overlap exceeds IOU th but also all object proposals, TIFF2025106196000014.tif10170 and Obtained that TIFF2025106196000015.tif is 10170, That is, since the object 110 is S 1,2 >S 1,1 it will still be correctly classified as "blue". For confirmation, after selecting the object proposal OP5 as the main object proposal for the object 112, (IOU 5,1 =0.02, IOU 5,2 =0.07, IOU 5,3 =0.03, IOU 5,4 =0.43, and IOU 5,6 =0.69, and after determining that according to the definition IOU 5,5 =1.0), Obtained that TIFF2025106196000016.tif is 10170 and TIFF2025106196000017.tif is 10170, or Obtained that TIFF2025106196000018.tif is 10170 and TIFF2025106196000019.tif is 10170 can also be confirmed, (Of course, including OP5) regardless of whether the iteration is performed over all object proposals or only over the object proposals OP4 and OP6 whose IOU exceeds IOU th confirm that since the object 112 is S 1,1 >S 1,2 it will be correctly classified as red.

[0081] As assumed in this specification, the ranking of the feature map layers can be performed manually, for example, based on the user experience, or can be performed in a more automated manner, for example. For example, it is assumed that the feature map layer scores (such as w l ) can be obtained by using an exhaustive search method to find the optimized weights / scores. For example, the score w lIf it is assumed that it should be between 0 and 1, the automated procedure starting from the assumption makes the score w l = 1 equal to all such scores. Then, for the lower resolution layer, the procedure can proceed by searching, for example, from 0.9 to 1.0 with an interval of 0.05 to find the score corresponding to the best average precision (AP) value or the like for the test dataset. As an example, hyperparameter optimization is used to search for a specific set of parameters (e.g., feature map layer weights / scores) that can result in optimal performance. Such optimization can be performed using one or more frameworks available for such purposes, such as Optuna (more detailed in Takuya Akiba et al., Optuna: A Next-generation Hyperparameter Optimization Framework, https: / / doi.org / 10.48550 / arXiv.1907.10902).

[0082] It is also assumed herein that a device, a computer program, and a computer program product for object attribute classification in an image are provided, as will be described in more detail below with reference to FIGS. 5A, 5B, and 6 as well.

[0083] FIG. 5A schematically shows device 500 for object attribute classification in an image, as previously described herein, i.e., a device configured to implement method 400, for example, as described with reference to FIG. 4. Device 500 includes at least a processor (or “processing circuit”) 510 and optionally a memory 512. As used herein, “processor” or “processing circuit” can be, for example, a suitable central processing unit (CPU), multiprocessor, microcontroller (μC), digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), graphics processing unit (GPU), etc., capable of executing software instructions stored in memory 512, and can be any combination of one or more of them. Memory 512 can be external to processor 510 or internal to processor 510. As used herein, “memory” can be any combination of random access memory (RAM) and read only memory (ROM), or any other type of memory capable of storing instructions. Memory 512 contains (i.e., stores) instructions that, when executed by processor 510, cause device 500 to implement the method described herein (i.e., method 400 or any embodiment of method 400). Device 500 can further include one or more additional items 514 that can be useful for implementing the method in some situations. In some exemplary embodiments, device 500 can be, for example, a (video) camera, such as a (video) surveillance camera, and the one or more additional items 514 can, in that case, include, for example, an image sensor and one or more lenses for focusing light from a scene onto the image sensor, and thus the surveillance camera can capture an image of the scene as part of implementing the envisioned method.(One or more) additional items 514 may also include various other electronic components required for, for example, operating an image sensor and / or lens appropriately as needed, and for gathering information related to the current orientation and / or focal length of the camera and (one or more) lenses, which may be used, for example, to determine the current FOV of the camera. Implementing this method in a surveillance camera can be useful in that the processing is moved to the “edge,” i.e., closer to where the actual scene is captured, compared to performing post-processing of object proposals elsewhere (such as in a more centralized processing server or the like). Device 500 may be connected to a network, for example, so that results from implementing this method can be sent to a user. For this purpose, device 500 may include a network interface 516, which may be, for example, a wireless network interface (defined in any of the IEEE 802.11 or subsequent standards, for example, supporting Wi-Fi) or a wired network interface (defined in any of the IEEE 802.3 or subsequent standards, for example, supporting Ethernet). Network interface 516 may also support any other wireless standard capable of transferring encoded video, such as, for example, Bluetooth or the like. The various components 510, 512, 514, and 516 (if present) may be connected via one or more communication buses 520, and thus these components can communicate with each other and exchange data as needed.

[0084] Device 500 can be a surveillance camera attached to or attachable to a building, for example, in the form of a PTZ camera, or, for example, a fisheye camera capable of providing a wider view of a scene, or any other type of surveillance / surveillance camera. Device 500 can be, for example, a body camera, an action camera, a dashcam, or the like, suitable for attachment to people, animals and / or various vehicles, or the like. Device 500 can be, for example, a smartphone or tablet that a user can carry and use to take pictures of a scene. In any such example of Device 500, it is assumed that Device 500 can include all necessary components (if any) other than those already described herein, as long as Device 500 is still capable of implementing Method 400 or any embodiment of Method 400 as contemplated herein. In some examples, the various components of Device 500 can be further configured to implement the ANN architectures / entities described herein, such as 200 or 201, for example. In other examples, Device 500 may simply be configured to receive the output from such an ANN architecture / entity and may only perform post-processing of multiple object proposals.

[0085] Figure 5B schematically shows one or more embodiments of device 500 with respect to several functional / calculation blocks 510a - 510d. Each such block 510a - 510d is responsible for implementing the functionality by specific operations of method 400 shown in the flowchart of FIG. 4. For example, one such functional block 510a may be configured to obtain an output from an ANN entity (operation S410), another block 510b may be configured to identify members of a first set (operation S420), another block 510c may be configured to rank (or obtain the ranking of) a feature map layer (operation S430), and another block 510d may be configured to determine an attribute class for a first attribute (operation S440). Device 500 may optionally include one or more additional functional blocks 510e, such as blocks for implementing an ANN architecture or the like.

[0086] Generally speaking, each functional module 510a - e can be implemented in hardware or in software. Preferably, one or more or all of the functional modules 510a - e can be implemented by processing circuit 510, optionally in cooperation with storage medium / memory 512 and / or communication interface 516. Processing circuit 510 is thus arranged to fetch instructions from memory 512 provided by functional modules 510a - e and execute these instructions, thereby performing any operations of method 400 implemented by / on device 500 as disclosed herein.

[0087] FIG. 6 schematically shows a computer program product 610 including a computer-readable medium / memory 630. A computer program 620 may be stored on the computer memory medium 630, and the computer program 620 may cause a processor 510, as well as entities and devices operatively coupled to the processor 510, such as communication interface 516 and memory 512 of device 500, to execute a method 400 according to the embodiments described with reference to FIGS. 2A, 2B, 3A, 3B, 3D, and 4. Thus, the computer program 620 and / or the computer program product 610 may provide means for performing any of the operations of method 400 implemented by device 500 as disclosed herein.

[0088] In the example of FIG. 6, the computer program product 610 and the computer-readable memory medium 630 are shown as an optical disk, such as a CD (compact disk) or a DVD (digital versatile disk) or a Blu-Ray disk. The computer program product 610 and the computer-readable memory medium 630 may also be embodied as a non-volatile memory medium of a device in an external memory, such as a memory, more specifically, a USB (universal serial bus) memory, or a flash memory such as a compact flash memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM). Thus, the computer program 620 is schematically shown here as a track on the optical disk shown, but the computer program 620 may be stored in any manner suitable for the computer program product 610 and the computer-readable memory medium 630.

[0089] In summary of the various embodiments presented herein, the present disclosure provides an improved way of post - processing object proposals from an ANN architecture / entity that utilizes feature maps and feature map layers for multiple spatial resolutions. In particular, the present disclosure proposes that just because a particular feature map (layer), for example, used to predict an object class is not particularly good at that task, the information in the object proposals should not be discarded, because the same feature map (layer) may be excellent at accurately classifying one or more attributes of the object at the same time. This is because the task of object detection (which is often based on identifying contours and shapes instead of things like color, texture, and the like) is often more suitably performed on lower - resolution images, while the task of attribute classification (such as for identifying color, texture, etc.) can be more suitably performed on higher - resolution images where such information (such as regarding color, texture, etc.) has not yet been lost. By ranking the feature map layers (with respect to the ability of the feature map layer for attribute classification), and taking into account how well each object proposal overlaps with the major object proposals (in terms of object location), the risk of misclassifying a particular attribute as belonging to the wrong attribute class can be reduced. Another advantage is that the proposed solution does not necessarily need to modify the existing ANN - based architecture already used to provide the object proposals, but instead can be implemented only as post - processing of such object proposals.

[0090] Features and elements may be described above in specific combinations, but each feature or element may be used alone, without other features and elements, or in various combinations with or without other features and elements. Additionally, variations of the disclosed embodiments can be understood and realized by those skilled in the art when practicing the claimed invention, from a study of the drawings, the present disclosure, and the appended claims.

[0091] In the claims, the words "comprising" and "including" do not exclude other elements, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that some features are described in mutually different dependent claims does not indicate that a combination of these features cannot be used advantageously.

Explanation of Signs

[0092] Image of 100 scenes 110, 112 Objects 200, 201 ANN architecture 210 Bottom-up path 212-j Convolution stage 214 Multiple network layers 216-j Feature map layer / stage 220 Top-down path 222-j Convolution stage 230 Head group 232 Object classification module 234 Object location identification module 240 Head group 242, 244 Attribute classification module 300 Data 320~326 Bounding box 330 Bounding box index 400 Method S410, 420, 430, 440 Operations 500 Device 510 Processing circuit 512 Memory 514 Additional circuit / function 516 Interface 510a~e Functional blocks 610 Computer program product 620 Computer program 630 Storage medium

Claims

Claim 1. A computer-implemented method (400) for object attribute classification in an image (100), comprising: - obtaining (S410) a plurality of object proposals in the same image from an output of an artificial neural network (ANN) entity (200, 201) trained to locate and classify objects (110, 112) in an image using a plurality of feature map layers (216-j) associated with different spatial resolutions, each object proposal indicating at least: i) an object class confidence score for a same first object class; ii) an estimated object location in the image; and iii) an attribute class confidence score for each of one or more different attribute classes for the same first attribute, each object proposal being associated with one of the feature map layers; - identifying (S420) among said plurality of object proposals and based on their respective indicated object class confidence scores a first set comprising a primary object proposal and one or more other object proposals; - ranking (S430) said feature map layers from least important to most important in terms of their ability to extract useful information for classification of said first attribute; determining an attribute class for the first attribute based on one or more attribute class confidence scores of all members of the first set (S440), including taking into account as part of said determining the attribute class for the first attribute the ranking of the feature map layer relative to the one or more other object proposals and object location overlap between the primary object proposal and the one or more other object proposals; Including, Determining the attribute class for the first attribute comprises: for each particular attribute class of said one or more attribute classes, iterating through the members of the first set, each of which indicates an attribute class confidence score for the particular attribute class; for each iteration, determining a term equal to or proportional to the product of an object location overlap size between the member of the first set and the leading object proposal, an overall ranking score for the feature map layer associated with the member, and the attribute class confidence score for the particular attribute class indicated by the member; and determining an overall attribute class score for the particular attribute class as equal to or proportional to the sum of the terms determined during said iterating; determining the attribute class for the first attribute as the attribute class having the highest overall attribute class score; The method (400) further comprises:

2. 2. The method of claim 1, wherein the object location of each object proposal is represented as a bounding box (320-326), and the method further comprises using an intersection / union (IOU) operation to determine the object location overlap between the primary object proposal (322) and the one or more other object proposals (321, 323-326).

3. 3. The method of claim 1, wherein taking into account the ranking and the object location overlap comprises giving attribute class confidence scores indicated by object proposals associated with feature map layers that have greater object overlap with the primary object proposal and that are ranked as more important greater importance for determining the attribute class of the first attribute than attribute class confidence scores indicated by object proposals associated with feature map layers that have less object overlap with the primary object proposal and that are ranked as less important.

4. The method of claim 1 or 2, wherein said iterating is performed only over the members of said first set that exhibit the highest attribute class confidence scores for said particular attribute class.

5. The method of claim 1 or 2, wherein the overall attribute class score for each particular attribute class is further defined as being inversely proportional to the number of members of the first set that are repeated.

6. 3. The method of claim 1, wherein the overall ranking score for the feature map layer associated with the member is defined as the ratio of the ranking score for the feature map layer to the sum of such ranking scores for all of the plurality of feature map layers.

7. The method of claim 1 or 2, wherein the ranking of the plurality of feature map layers depends on the first attribute.

8. The method of claim 1 or 2, further comprising: using a non-maximum suppression (NMS) operation to distinguish between the primary object proposal and the one or more other object proposals.

9. The method of claim 1 or 2, wherein the ANN entity comprises a Feature Pyramid Network (FPN) (220) for providing the plurality of feature map layers.

10. The method of claim 1 or 2, wherein the ANN entity comprises a plurality of convolutional layers (210) for providing the different spatial resolutions of the image.

11. A device (500) for object attribute classification in an image (100), said device comprising a processing circuit (510), said processing circuit (510) providing said device with: obtaining a plurality of object proposals in the same image from the output of an artificial neural network (ANN) entity (200, 201) trained to locate and classify objects (110, 112) in an image (100) using a plurality of feature map layers (216-j) associated with different spatial resolutions, each object proposal indicating at least: i) an object class confidence score for a same first object class; ii) an estimated object location in the image; and iii) an attribute class confidence score for each of one or more different attribute classes for the same first attribute, each object proposal being associated with one of the feature map layers; - identifying a first set comprising a primary object proposal (322) and one or more other object proposals (321, 323-326) among said plurality of object proposals and based on their respective indicated object class confidence scores; obtaining a ranking of said feature map layers in terms of their ability to extract useful information for classification of said first attribute, from least important to most important; determining an attribute class for the first attribute based on one or more attribute class confidence scores of all members of the first set, including taking into account as part of said determining the ranking of the feature map layer relative to the one or more other object proposals and object location overlap between the primary object proposal and the one or more other object proposals; configured to cause Determining the attribute class for the first attribute comprises: for each particular attribute class of said one or more attribute classes, iterating through the members of the first set, each of which indicates an attribute class confidence score for the particular attribute class; for each iteration, determining a term equal to or proportional to the product of an object location overlap size between the member of the first set and the leading object proposal, an overall ranking score for the feature map layer associated with the member, and the attribute class confidence score for the particular attribute class indicated by the member; and determining an overall attribute class score for the particular attribute class as equal to or proportional to the sum of the terms determined during said iterating; determining the attribute class for the first attribute as the attribute class having the highest overall attribute class score; The device (500) further comprises:

12. The device of claim 11, wherein the processing circuitry is further configured to cause the device to implement the ANN entity (200, 201).

13. A computer program (620) for object attribute classification in an image (100), the computer program, when executed by a processing circuit (510) of a device (500), causing the device to: obtaining a plurality of object proposals in the same image from the output of an artificial neural network (ANN) entity (200, 201) trained to locate and classify objects in the image (100) using a plurality of feature map layers (216-j) associated with different spatial resolutions, each object proposal indicating at least i) an object class confidence score for a same first object class, ii) an estimated object location in the image, and iii) an attribute class confidence score for each of one or more different attribute classes for the same first attribute, each object proposal being associated with one of the feature map layers; - identifying a first set comprising a primary object proposal (322) and one or more other object proposals (321, 323-326) among said plurality of object proposals and based on their respective indicated object class confidence scores; obtaining a ranking of said feature map layers in terms of their ability to extract useful information for classification of said first attribute, from least important to most important; determining an attribute class for the first attribute based on one or more attribute class confidence scores of all members of the first set, including taking into account as part of said determining the ranking of the feature map layer relative to the one or more other object proposals and object location overlap between the primary object proposal and the one or more other object proposals; configured to cause Determining the attribute class for the first attribute comprises: for each particular attribute class of said one or more attribute classes, iterating through the members of the first set, each of which indicates an attribute class confidence score for the particular attribute class; for each iteration, determining a term equal to or proportional to the product of an object location overlap size between the member of the first set and the leading object proposal, an overall ranking score for the feature map layer associated with the member, and the attribute class confidence score for the particular attribute class indicated by the member; and determining an overall attribute class score for the particular attribute class as equal to or proportional to the sum of the terms determined during said iterating; determining the attribute class for the first attribute as the attribute class having the highest overall attribute class score; The computer program (620) further comprises:

14. A computer program product (610) comprising a computer readable storage medium (630) storing a computer program (620) according to claim 13.

Images