Classifying objects in images

Abstract

The present invention relates to image classification, and more particularly to object recognition / detection in images. According to an aspect of the present invention, a computer-implemented method for classifying an object in an image is provided. The method includes: identifying a plurality of local features of an object in the image using one or more machine-learned local feature identification models, where each local feature has a respective feature identity and a respective feature area; comparing the identified local features with a plurality of sets of masked images of candidate objects, where each set of masked images corresponds to a respective candidate object and comprises a plurality of masked images of the respective candidate objects from a plurality of viewing angles, where each masked image has a plurality of feasible search zones for the local features of the candidate objects; and classifying the object in the image as one of the candidate objects based on the comparison of the identified local features with the plurality of sets of masked images of the candidate objects.

Description

[Technical Field]

[0001] The present invention relates to image classification, and in particular to object recognition / detection in images. [Background technology]

[0002] Current state-of-the-art deep learning methods can be described as opaque (unexplainable), brittle (not robust to scene or object variances and susceptible to adversarial attacks), computationally intensive to train (hence difficult to manage when frequent updates and upgrades are required), and greedy (often requiring hundreds to millions of images for training). This can make them unsuitable for applications in contexts requiring high levels of model explainability, robustness, and manageability, such as high object classification and detection accuracy, high object class variance (e.g., color changes, camouflage, and the addition of new external features), and defense. These challenges can be exacerbated by the limited availability of image data, especially when new object classes need to be added to the model. Summary of the Invention

[0003] According to an aspect of the present invention, a computer-implemented method for classifying an object in an image is provided, comprising: identifying a plurality of local features of an object in the image using one or more machine-learned local feature identification models, where each local feature comprises a respective feature identity and a respective feature area; comparing the plurality of identified local features to a plurality of sets of masked images of candidate objects, where each set of masked images corresponds to a respective candidate object and comprises a plurality of masked images of the respective candidate objects from a plurality of viewing angles, where each masked image comprises a plurality of feasible search zones for the local features of the candidate objects, where the comparison of the plurality of local features to the masked images of the candidate objects is based on overlap between the identified local features and the feasible search zones of the candidate object in the masked images; and classifying the object in the image as one of the candidate objects based on the comparison of the identified local features to the plurality of sets of masked images of the candidate objects.

[0004] The comparison of the plurality of local features with the masked image of the candidate object may be based on a weighted sum of the overlap scores. The weights in the weighted sum may be based at least in part on feature weights indicating the relative importance of each local feature type. Identifying the plurality of local features of the object in the image may further comprise determining a confidence score for each of the local features. The weights in the weighted sum may be based at least in part on the confidence scores for identified ones of the local features.

[0005] The comparison of the multiple local features with the masked image of the candidate object may be based on an intersection over union measure.

[0006] The feature area associated with each local feature may comprise the bounding box of the local feature.

[0007] Each set of masked images may comprise multiple masked images of each candidate object at multiple viewing distances.

[0008] The method may further comprise selecting a plurality of sets of masked images of the candidate objects from a superset of the sets of masked images of the candidate objects based on the identity of local features identified in the images.

[0009] The method may further comprise constructing a three-dimensional model of the further candidate object; generating from the three-dimensional model a set of masked images of the further candidate object, the set comprising a plurality of masked images of the further candidate object, each masked image comprising a plurality of local features of the further candidate object, each local feature comprising a respective feature identity and a respective feature area; and adding the set of masked images of the further candidate object to the plurality of sets of masked images of the candidate object.

[0010] The one or more machine-learned local feature identification models may comprise a plurality of neural networks, each of which may be configured to identify local features of a respective feature identity type. The one or more machine-learned local feature identification models may comprise a plurality of region-based convolutional neural networks.

[0011] The candidate object may comprise a land vehicle, and the local feature identities for the land vehicle may comprise one or more of a wheel, a windshield, a turret, a weapon system, an exhaust, a track, a bumper, and / or an antenna.

[0012] The candidate object may comprise an aircraft, and the local feature identities for the aircraft may comprise one or more of a wing, a landing gear, an engine, a canopy, a tail, a nose, a jet, a pod, a weapon system, and / or a canard.

[0013] The candidate object may comprise a maritime vehicle, and the local feature identities for the maritime vehicle may comprise one or more of a superstructure element, a turret, a smokestack, a weapons system, a bridge, a radar array, and / or a helipad.

[0014] The method may further comprise estimating an object scale of the object in the image and scaling the masked image of the candidate object based on the estimated object scale. Estimating the object scale of the object in the image may be based on one or more background or foreground objects in the image. Estimating the object scale of the object in the image may be based on a known geographic location in the image.

[0015] Identifying a plurality of local features of an object in an image may comprise identifying a plurality of local feature locations in the image using one or more machine-learned local feature identification models, clustering the identified plurality of local features in the image using a clustering algorithm, and selecting the clusters of the identified local features as a plurality of local features of the object in the image.

[0016] According to a further aspect of the present invention, there is provided a computer program product comprising computer readable instructions that, when executed by a computer, cause the computer to perform any one or more of the methods described herein.

[0017] According to a further aspect of the present invention, there is provided an apparatus comprising one or more processors and a memory, the memory storing computer-readable instructions that, when executed by the one or more processors, cause the apparatus to perform any one or more of the methods described herein.

[0018] According to a further aspect of the present invention, there is provided a system comprising one or more imaging sensors, one or more processors, and a memory, the memory storing computer-readable instructions that, when executed by the one or more processors, cause the apparatus to perform any one or more of the methods described herein.

[0019] Embodiments of the present invention will now be described, by way of example only, with reference to the drawings, in which: [Brief explanation of the drawings]

[0020] [Figure 1] 1 shows a schematic diagram of a method for object recognition / detection in images. [Figure 2] An example of determining an intersection-over-union measure is given below. [Figure 3] We present an overview of low-shot learning and execution methods for updating image classification models. [Figure 4] 1 shows a flow diagram of an example method for identifying objects in an image. [Figure 5] 1 shows a schematic diagram of a computer system / apparatus for performing any of the methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0021] This specification describes systems, devices, and methods for object recognition in images. The object recognition techniques described herein use a combination of machine-learned feature detection models and masked images generated from 3D object models to identify objects in images. Local features of the object in the image and their locations are identified using one or more machine-learning models. These identified local features are compared to known feature identities and locations in a set of masked images of the object based on how much they overlap to determine the candidate object that best matches the identified local features.

[0022] The systems, devices, and methods described herein follow a hybrid approach, which in some embodiments may be referred to as neuro-geo-symbolic. The front end of what is a multi-stage process is performed by one or more machine learning models, which identify local features of interest in an image and generate predicted bounding boxes and, in some embodiments, confidence values ​​over relevant portions of a test or target image. In some embodiments, a federated network of region-based convolutional neural networks (R-CNNs) may be used to detect local features, each of which may be trained to detect a particular local feature type (e.g., wheels, weapons, bumpers, windshields, fuel tanks, etc.).

[0023] The geometric and symbolic stages effectively evaluate whether the predicted local features are in the correct location relative to each other by a cumulative scoring process using a set of 3D rendering tool-generated segmented image masks (which serve as "feasible search zones") for each object class across a set of pose (or observation) angles and, in some embodiments, scales. The score is based on the overlap between the predicted local feature in the image and the feasible search zone of the local feature in the segmented image mask. The score, in some embodiments, may be based on an IOU-confidence product sum for each determined or predicted local feature, where IOU is an intersection-over-union measure (i.e., comparing what was predicted by the deep learning stage with the prediction represented by the segmented image mask). These may be added together to form an overall score for each segmented image mask. This results in a ranking table where the highest scores represent the predicted object class and predicted object pose angle.

[0024] In this way, objects in an image can be identified in an explainable manner that is robust to adversarial attacks, high object class variance, scale, and partial occlusions.

[0025] Attack vectors such as adversarial noise and images may have limited impact on the techniques described herein for several reasons. First, the adversarial noise would need to affect several feature detector models (e.g., multiple R-CNNs in a federated network) at the local feature level within the image. This would be a difficult task for an adversary, even if they had access to trained models, and would likely still be outdone by the mathematics in the process. Second, an adversary could introduce adversarial images as markings on an object (e.g., a vehicle), but they would be effectively outdone by the "real" local features and would not align with expectations (represented by the segmented image masks used in the process).

[0026] High intra-class object variance can also be accommodated within the techniques described herein. For example, defensive objects of interest may be applied with different colors, camouflage, weapons, and other topological manifolds, and may be covered in intra-class dirt and debris. For example, a particular type of main battle tank may be fitted with fences (i.e., counter-RPGs and counter-anti-armor weapons), may be fitted with small arms, and / or may be in desert camouflage rather than urban camouflage. This intra-class variance poses a significant challenge for deep learning image classifiers (pixel correlators) and significantly increases the need for large training datasets. However, because the models described herein focus on local features and their relative positions that change little despite the aforementioned causes, a degree of robustness may exist.

[0027] The effects of partial occlusion of objects in an image can also be reduced. When an object is partially occluded, important local features are still visible and can be detected, and high-accuracy classification can still be performed. This is because the rankings and calculations are all relative, based on what can be observed rather than suffering from what is missing and what is expected. In contrast, the occlusion problem is expected to pose significant challenges for existing deep neural network classifiers.

[0028] The techniques described herein can also facilitate low-shot learning to incorporate new objects into a classifier without requiring large datasets of examples of the new objects or costly retraining of machine learning models. This can occur as a result of a deep learning stage that is not directly involved in classifying objects. It is the geo-symbolic phase that classifies observed objects. This allows for a simple solution that can classify new object classes as long as the classification is determined by the presence or absence of distinctive local features and their expected relative locations, and does not require the retraining of large, greedy machine learning models.

[0029] This can be achieved as follows: A user takes a single new real-world image, or a small number of images, of a new object class, constructs a 3D representation of it, and segments local features, e.g., by assigning each one a unique RGB value. This is a relatively simple process that takes a strong amateur modeler at best tens of minutes. Using a 3D modeling tool and a predefined camera path, a dataset of segmented image masks can be generated for this new class and added to the overall dataset of masks. This can be achieved with a runtime of less than a few minutes. Any new images subsequently provided for analysis / classification containing the new class will be successfully classified because they are now represented in the dataset used by the geo-symbolic stage of the model. What this achieves is a process for providing classification based on what is attainable with one-shot or few-shot learning capabilities.

[0030] Furthermore, some of the methods described herein enable a high degree of parallelization, improving the speed of object classification during inference. Using a federated approach with multiple machine learning models, the local feature identification stage can be performed across parallel processing units, thus speeding up local feature detection. This also applies to geo-symbolic models, as they can be easily parallelized as long as each computing element can be given one or more of the segmented image masks for computing ranking scores.

[0031] 1 shows a schematic diagram of a method 100 for object recognition / detection in images. The method may be performed by one or more computers operating in one or more locations.

[0032] An image 102 containing an object 104 to be identified is input into one or more machine-learned local feature identification models 106. The one or more machine-learned local feature identification models 106 process the input image 102 to identify one or more local features 108A-C present in the image and their respective locations. The identified local features 108A-C and their corresponding locations are output from the one or more machine-learned local feature identification models 106. In some embodiments, a confidence score for each identified local feature is also output by the one or more machine-learned local feature identification models 106. The identified local features 108A-C are compared to masked images of candidate objects in a plurality of sets 110A-M of masked images to determine a classification output 112 for the input image 102 that indicates the object type of the object 104 in the input image 102.

[0033] The one or more machine-learned local feature identification models 106 may comprise multiple local feature identification models 106A-N, each specialized (e.g., trained) for detecting and locating local features of a predefined type. Each model 106 may take as input the input image 102 or a region of the input image and identify the presence / absence of local features of the type associated with the model. If local features of that type are present, their locations are also identified.

[0034] The one or more machine-learned local feature identification models 106 may comprise one or more neural network models. The one or more neural network models may comprise one or more region-based convolutional neural networks (R-CNNs). R-CNNs are neural networks that combine region proposals with convolutional neural network features. In some embodiments, multiple R-CNNs are used, each trained to detect (e.g., identify and localize) local features of a particular predefined local feature class. In some implementations, image registration may be used to center objects in an image.

[0035] The local features 108A-C comprise predefined classes of sub-features of the object (also referred to herein as "predefined local feature classes"). For example, if the method is being used to identify vehicles in an image, the local features may comprise elements of the vehicle. For example, for a land vehicle, the local feature classes may comprise wheels, windshield, windows / viewing slits, exhausts, tracks, weapon systems, turrets, engine grills, antennas, or the like. For a marine vehicle, the local feature classes may comprise smokestacks, hull features, bridges, antennas, radar arrays, weapon systems, landing pads, or the like. For an aircraft, the local feature classes may comprise engines, air intakes, nose cones, weapon mounts, weapon systems, tails, wings, fins, cockpits, or the like.

[0036] Each local feature 108A-C identified in the image 102 is associated with a corresponding feature identity and feature location. The feature identity identifies which of a plurality of predefined local feature classes the local feature 108A-C belongs to. The feature location indicates the area of ​​the image in which the associated local feature is found. It may, for example, be in the form of a bounding box around the identified local feature. Alternatively, the outline of the identified local feature may be used as the feature location, or the center point of the feature may be identified as the feature location.

[0037] The identified local features 108A-C are then compared to local features in a set of masked images 110. Each set of masked images 110A-M comprises masked images of the candidate object from multiple different viewing angles, and in some embodiments, at different scales / viewing distances. Each set of masked images 110A-M may be derived from a respective three-dimensional model of the object to which it corresponds. The comparison for each identified local feature 108A-C may, in some embodiments, be performed in parallel. Furthermore, each masked image and / or set of masked images may be processed in parallel.

[0038] Each masked image comprises a set of feasible search zones for local features of objects in the image. Local feature types may be coded in the masked image using RGB coding, with each feature type assigned a respective color in the masked image, and feasible search zones for local features colored with their respective feature colors in the masked image. For example, feasible search zones for each wheel-type object may be colored red in the masked image, and feasible search zones for each window-type object may be colored yellow.

[0039] A comparison of the identified local features 108A-C with the masked images of the candidate objects in the plurality of sets of masked images 110 may be performed based on the overlap between the feature locations of the identified local features 108A-C and the feasible search zones defined by the local features in the masked images 110. For example, if the feature locations of the identified local features 108A-C are defined by bounding boxes, a measure of overlap between the bounding boxes and the feasible search zones in the masked images may be used to perform the comparison.

[0040] An example of such a measure is the Intersection Over Union (IOU) measure. The IOU measure for two objects is given by dividing the overlapping area between the two objects by the total area covered by both objects, i.e., the overlapping area between the two objects plus the non-overlapping area. Figure 2 shows an illustrative method 200 for determining the IOU of a local feature (in this example, a wheel) 202 identified in an input image 204 using a feasible search zone 206 in a masked image 208. The local feature 202 in the input image 204 has a feature location defined by a bounding box. The feasible search zone 206 in the masked image 208 is defined by the outline of the region where the local feature may reside.

[0041] The area of ​​intersection 210 is determined by taking the intersection between the area of ​​the identified local feature 202 and the feasible search zone 206 in the masked image. The area of ​​union 212 is determined by taking the union between the area of ​​the identified local feature 202 and the feasible search zone 206. The IOU between the local feature 202 and the feasible search zone 206 is given by dividing the area of ​​intersection 210 by the area of ​​union 212. Mathematically, the IOU, I, between two objects is given by:

[0042]

number

[0043] where A o is the area of ​​intersection between two objects, and A u is the area of ​​the union between two objects. A perfect score of 1 is achieved when two objects exactly overlap.

[0044] Returning to Figure 1, each local feature prediction for a particular local feature type is compared with the corresponding FSZ (for the same local feature class) and an IOU is calculated. Comparison of the identified local features 108A-C with the masked image of the candidate object is performed based on the sum of these IOUs.

[0045] The comparison may, in some embodiments, be performed using a weighted sum of the IOUs of the identified local features 108A-C in the image 102. The weighting may be based on the confidence scores associated with the identified local features output by the machine learning model 106. For example, for a local feature type i, the weighted IOU sum for that feature type may be given by:

[0046]

number

[0047] where I j is the IOU score for the j-th local feature 108A-C, and C j is the corresponding confidence score for the jth local feature, and m is the total number of local features of type i identified in the image. The overall score T for the candidate object c c can be calculated as the sum of this weighted sum over all local feature classes i.

[0048] Alternatively or additionally, the weighting may be a feature weighting W i Feature weighting assigns relative importance to each class of local feature. For example, given the ubiquity of wheels on vehicles, a wheel-type local feature may be given a lower weight than, say, a weapon-type local feature. If confidence weighting is also used, the weighted sum of IOUs for candidate object c may be given by:

[0049]

number

[0050] where W i is the feature weighting, and S i is the confidence-weighted feature sum, and i labels the feature class.

[0051] The classification output 112, C, may comprise the identity of the candidate object having the highest scoring masked image among the plurality of sets of masked images 110. Alternatively, the classification output 112 may comprise a list of the top N scoring candidate objects, where N may be, for example, between 5 and 20, e.g., 10.

[0052] In some implementations, feature logic may be applied prior to comparing the identified local features 108 to the multiple sets of masked images 110. The feature logic eliminates candidate objects from the comparison based on the identified local features 108. For example, if a particular feature is identified as present, the sets of masked images 110 corresponding to objects that do not have that feature are not used in the comparison. In practice, a subset comprising the sets of masked images of objects that have the identified feature is identified and used for the comparison.

[0053] The feature logic may also consider the number of local features of a particular type identified in the image: for example, if six wheels are identified, only the set of masked images with objects with at least six wheels will be used for comparison.

[0054] The combination of the feature logic and feature comparison process may be referred to herein as a "geo-symbolic model."

[0055] In some implementations, local features may be identified that do not belong to the object 104, for example, that belong to other objects present in the image. To address this, a clustering algorithm may be applied to the identified local features in the image 102 to collect them into multiple clusters. One or more of these clusters may be chosen to be local features of the object to be identified.

[0056] For example, if the objects 104 in the image 102 are separated (i.e., do not overlap each other), clustering methods such as k-nearest neighbors or SVM based on the relative proximity of local feature prediction bounding boxes can be utilized. A group of local feature predictions within close proximity to each other can be considered a cluster and, therefore, considered to belong to a single object. This prediction data is utilized by a post-registration process of the geo-symbolic model, where each cluster of predictions is addressed separately because it corresponds to a different vehicle. This method can mitigate the problem of false-positive local feature predictions caused by complex and cluttered environmental backgrounds, since single local feature predictions or local feature predictions of only the same class can be ignored.

[0057] The clustering method can be further extended by knowing the distribution of object sizes in the image in terms of pixels. For example, some feature predictions can be discarded because they are located too far from the centroid of the local feature prediction to be considered to belong to the object in question (i.e., the object being classified). From this, they should belong to another object or are simply false positive detections from features in the local environment. The threshold distance from the object centroid used to determine which local features to discard can be based on the largest size of the candidate objects in the set of candidate objects that match the identified local feature.

[0058] Discarding such local features can be achieved through knowledge of the distance subtended by an image pixel at the object location, considering that the resolution and field of view of the imaging sensor are known and, from this, the instantaneous field of view is known via calculation (i.e., the instantaneous field of view is the field of view projected from a single detector pixel corresponding to the image pixel; therefore, the total field of view of the sensor is simply divided by the detector resolution in one of the detector dimensions). The pointing direction of the imaging sensor may also be known (e.g., through an angle converter on the sensor assembly, if not fixed in a fixed pointing direction relative to the platform, and through the platform position and attitude arrived at through the navigation and guidance system). The range between the platform and the object can then be determined using the intersection of the digital terrain model and the sensor's line of sight (boresight). The distance corresponding to a pixel can be approximately calculated in each horizontal and vertical dimension by appealing to small-angle theory (i.e., range in radians × IFOV angle). From this, the distance between local features in the image can be estimated.

[0059] Figure 3 shows an overview of a low-shot learning method 300 for updating an image classification model. One or more images 302 of an object corresponding to a new object class are acquired. From these images 302, a three-dimensional model 304 of the object in the image is generated. The three-dimensional model may be composed of primitives, at least some of which may correspond to local features of the object. Features in the three-dimensional model that are not present in the image itself, such as the location of a wheel that is obscured in the image, may be inferred. Symmetry considerations may be used to infer the location and / or type of such features.

[0060] Local features 306 in the three-dimensional model 304 are identified. This can be done manually using a feature identification model or by construction (e.g., using primitives that correspond to the local features, as described above). The local features can be color-coded, for example, by feature type. The color coding can be based on a lookup table.

[0061] Based on the 3D model 304 and the identified local features 306, multiple masked images 308 of the object are generated, each corresponding to a 2D view of the 3D model from a different angle and / or at a different scale / viewing distance. Each masked image 308 shows a feasible search zone for the object's local features when the object is viewed from that angle and / or distance.

[0062] The plurality of masked images 308 are then stored in memory / database 310 as a new set of masked images for use in object classification, as described above in connection with FIG.

[0063] In this way, newly observed objects can be added to the image classification method based on only one image (or a few images) of the object in the new image class, and expensive retraining of the deep learning model is not required.

[0064] Figure 4 shows a flow diagram of an illustrative method for identifying objects in an image. The method may be performed by a computing system such as the system described in connection with Figure 5. The method corresponds to the method described above in connection with Figures 1-3.

[0065] In operation 402, multiple local features of an object in an image are identified using one or more machine-learned local feature identification models (e.g., multiple neural networks such as R-CNN). Each local feature is associated with a respective feature identity and a respective feature area (i.e., its location in the image).

[0066] Each machine-learned local feature identification model takes an image or a portion of an image as input and processes it according to a set of learned parameters to generate an output indicative of feature identities and areas. Each machine-learned local feature identification model may be specialized to detect a particular type of feature.

[0067] The one or more machine-learned local feature identification models may be neural networks, for example, R-CNN.

[0068] The feature area may be represented as a bounding box around the identified local feature. Alternatively, the feature outline may be used.

[0069] In some implementations, a confidence score for each identified local feature may also be output from one or more machine-learned local feature identification models.

[0070] In some implementations, local features that do not belong to the object may be identified. To reduce these effects, a clustering algorithm may be applied to cluster the identified local features, and a subset of the clusters may be selected to be local features of the object. The identification of local features that do not belong to the object may further be based on knowledge of the delimited pixel areas for all object classes of interest, as described above in connection with FIG. 2.

[0071] In operation 404, the plurality of identified local features are compared to a plurality of sets of masked images of the candidate objects. Each set of masked images corresponds to a respective candidate object and comprises a plurality of masked images of the respective candidate object from a plurality of viewing angles. Each masked image comprises a plurality of feasible search zones of local features of the candidate object, each local feature comprising a respective feature identity and a respective feature area (i.e., its location in the image). The comparison is based on overlap between the identified local features in the image and the local features of the candidate object in the masked images.

[0072] The overlap between the identified local features in the image and the local features of the candidate object may be based on an intersection-over-union measure, as described above in connection with FIG. 2. A weighted sum of the intersection-over-union measures may be taken for each masked image to generate a score for the image indicating how well the image matches the identified features. The weights in the sum may be based on one or more of the confidence values ​​for each identified local feature and / or feature weights for each local feature type, where the feature weight indicates the relative importance of the corresponding feature type. This may increase the explainability of the class prediction, because the contribution of each IOU, and hence the local feature, is known and can be presented to the user.

[0073] In some implementations, the multiple sets of masked images of candidate objects are selected from a larger set (superset) of sets of masked images of candidate objects based on the identified local features. For example, if a particular type of local feature is identified as being present, sets of masked images corresponding to objects that do not have that local feature may be excluded from the comparison. This can increase the computational efficiency of the method, resulting in faster classification and / or using fewer computational resources (e.g., memory, processing power) to perform the classification.

[0074] In some implementations, the masked image of the candidate object may be scaled based on an estimated object scale prior to performing the comparison. The object scale may be estimated from characteristics of the input image, such as a known size (or size range) of the object in the background or foreground of the image. Alternatively or additionally, the object scale may be estimated based on a known geographic location of the image, for example, by comparing the object to the positions and / or sizes of known geographic features in the image.

[0075] In operation 404, an object in the image is classified as one of the candidate objects based on a comparison of the identified local features with a plurality of sets of masked images of the candidate objects. The identification may be the identity of the candidate object whose masked image best matches the identified local features in the image (e.g., has the highest score based on a weighted sum of overlap scores). Alternatively, the identification may be in the form of a set of scores for the candidate object indicating how closely the candidate object matches the local features in the input image.

[0076] The candidate object may comprise a land vehicle. The local feature identity for the land vehicle may comprise one or more of a wheel, a windshield, a turret, a weapon system, an exhaust, a track, a bumper, and / or an antenna. The candidate object may comprise an aircraft. The local feature identity for the aircraft may comprise one or more of a wing, a landing gear, an engine, a canopy, a tail, a nose, a jet, a pod, a weapon system, a propeller, and / or a canard. The candidate object may comprise a maritime vehicle. The local feature identity for the maritime vehicle may comprise one or more of a superstructure element, a turret, a funnel, a weapon system, a bridge, a radar array, a propeller, and / or a helipad.

[0077] If the object in the image does not correspond to a known candidate object, the classification method may be updated based on the image. A three-dimensional model of the further candidate object may be constructed based on the object in the image. From the three-dimensional model, a set of masked images of the further candidate object is generated. The set comprises a plurality of masked images of the further candidate object, each masked image comprising a plurality of local features of the further candidate object, each local feature comprising a respective feature identity and a respective feature area. The set of masked images of the further candidate object is then added to the plurality of sets of masked images of the candidate object.

[0078] 5 shows a schematic diagram of a computer system / apparatus 500 for performing any of the methods described herein. The system / apparatus 500 may be a distributed system.

[0079] The device (or system) 500 includes one or more processors 502. The one or more processors control the operation of other components of the system / device 500. The one or more processors 502 may include, for example, a general-purpose processor. The one or more processors 502 may be single-core or multi-core devices. The one or more processors 502 may include a central processing unit (CPU) or a graphics processing unit (GPU). Alternatively, the one or more processors 502 may include dedicated processing hardware, for example, programmable hardware with embedded firmware or a RISC processor. Multiple processors may be included.

[0080] The system / device includes a memory 504. The memory 504 may include working memory or volatile memory. One or more processors may access the volatile memory to process data and may control the storage of data in the memory. The volatile memory may include any type of RAM, for example, static RAM (SRAM), dynamic RAM (DRAM), or flash memory, such as an SD card.

[0081] The memory 504 may further comprise a non-volatile memory that stores a set of operating instructions in the form of computer-readable instructions for controlling the operation of the processor 502. The non-volatile memory may be any type of memory, such as a read-only memory (ROM), a flash memory, or a magnetic drive memory.

[0082] The one or more processors 502 are configured to execute operational instructions to cause the system / device to perform any of the methods described herein. The operational instructions may comprise code (i.e., drivers) associated with hardware components of the system / device 500 and code associated with basic operations of the system / device 500. Generally speaking, the one or more processors 502 execute one or more of the operational instructions stored permanently or semi-permanently in non-volatile memory and use volatile memory to temporarily store data generated during execution of the operational instructions.

[0083] Implementations of the methods described herein may be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These may include a computer program product (e.g., software stored on a magnetic or optical disk, memory, programmable logic device, etc.) comprising computer-readable instructions that, when executed by a computer, cause the computer to perform one or more of the methods described herein, such as those described in connection with FIG.

[0084] Any system features as described herein may also be provided as method features, and vice versa. As used herein, means-plus-function features may alternatively be expressed in terms of their corresponding structure. In particular, method aspects may be applied to system aspects, and vice versa.

[0085] Furthermore, any, some, and / or all features of one aspect may be applied to any, some, and / or all features of any other aspect, in any appropriate combination. It should also be recognized that specific combinations of the various features described and defined in any aspect of the invention may be implemented and / or provided and / or used independently.

[0086] While several embodiments have been shown and described, it will be recognized by those skilled in the art that changes can be made in these embodiments without departing from the principles of the present disclosure, the scope of which is defined in the claims.

Claims

1. 1. A computer-implemented method for classifying objects in an image, the method comprising: identifying a plurality of local features of an object in the image using one or more machine-learned local feature identification models, wherein each local feature comprises a respective feature identity and a respective feature area; comparing the identified local features to a plurality of sets of masked images of candidate objects; each set of masked images corresponds to a respective candidate object and comprises a plurality of masked images of said respective candidate object from a plurality of viewing angles; each masked image comprising a plurality of feasible search zones for local features of said candidate object; the comparing the plurality of local features with a masked image of the candidate object is based on an overlap between the identified local features and a feasible search zone of the candidate object in the masked image; classifying the object in the image as one of the candidate objects based on the comparison of the identified local features with a plurality of sets of masked images of candidate objects; A method comprising:

2. The method of claim 1 , wherein the comparison of the plurality of local features with a masked image of a candidate object is based on a weighted sum of overlap scores.

3. The method of claim 2 , wherein the weights in the weighted sum are based at least in part on feature weights indicating the relative importance of each local feature type.

4. identifying the plurality of local features of the object in the image further comprises determining a confidence score for each of the local features; a weight in the weighted sum based at least in part on the confidence score for the identified one of the local features; The method according to claim 2 or 3.

5. The method of any one of claims 1 to 4, wherein the comparison of the plurality of local features with a masked image of the candidate object is based on an intersection-over-union measure.

6. The method of any one of claims 1 to 5, wherein the feature area associated with each local feature comprises a bounding box of the local feature.

7. The method of any one of claims 1 to 6, wherein each set of masked images comprises a plurality of masked images of the respective candidate object at a plurality of viewing distances.

8. 8. The method of claim 1, further comprising selecting the plurality of sets of masked images of candidate objects from a superset of sets of masked images of candidate objects based on the identities of local features identified in the images.

9. constructing a three-dimensional model of the further candidate object; generating a set of masked images of the further candidate object from the three-dimensional model, the set comprising a plurality of masked images of the further candidate object, each masked image comprising a plurality of local features of the further candidate object, each local feature comprising a respective feature identity and a respective feature area; adding the set of masked images of the further candidate object to the plurality of sets of masked images of candidate objects; The method of any one of claims 1 to 8, further comprising:

10. 10. The method of claim 1, wherein the one or more machine-learned local feature identification models comprise a plurality of neural networks, each neural network in the plurality of neural networks configured to identify local features of a respective feature identity type.

11. The method of claim 10 , wherein the one or more machine-learned local feature discrimination models comprise multiple region-based convolutional neural networks.

12. 12. The method of claim 1, wherein the candidate objects comprise land vehicles and the local feature identities for the land vehicles comprise one or more of wheels, windshields, turrets, weapon systems, exhausts, tracks, bumpers, and / or antennas.

13. 13. The method of any one of claims 1 to 12, wherein the candidate objects comprise aircraft, and the local feature identities for the aircraft comprise one or more of wings, landing gear, engines, canopy, tail, nose, jets, pods, weapon systems, and / or canards.

14. 14. The method of any one of claims 1 to 13, wherein the candidate objects comprise a maritime vehicle and the local feature identities for the maritime vehicle comprise one or more of a superstructure element, a turret, a chimney, a weapons system, a bridge, a radar array, and / or a helipad.

15. estimating an object scale of the object in the image; scaling the masked image of the candidate object based on the estimated object scale; The method of any one of claims 1 to 14, further comprising:

16. The method of claim 15 , wherein estimating the object scale of the object in the image is based on one or more background or foreground objects in the image.

17. The method of claim 15 , wherein estimating the object scale of the object in the image is based on a known geographic location of the image.

18. Identifying the plurality of local features of an object in the image comprises: identifying a plurality of local feature locations in the image using the one or more machine-learned local feature identification models; clustering the identified local features in the image using a clustering algorithm; selecting the identified clusters of local features as the plurality of local features of the object in the image; The method of any one of claims 1 to 17, comprising:

19. A computer program product comprising computer readable instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 18.

20. 19. An apparatus comprising one or more processors and a memory, the memory storing computer readable instructions that, when executed by the one or more processors, cause the apparatus to perform a method according to any one of claims 1 to 18.

21. 1. A system comprising: one or more imaging sensors; one or more processors; Memory and wherein the memory stores computer readable instructions that, when executed by the one or more processors, cause the system to perform a method according to any one of claims 1 to 18.

Images