Processing images of objects and object parts, including multi-object constructs and deformed objects

Abstract

A method of image processing may include receiving an image of an instance of an object and a three-dimensional model of the object, detecting a first plurality of landmarks of the instance of the object in the two-dimensional image, estimating a pose of the instance of the object in the received image relative to an imaging device that captured the image, where a relative pose in the received image is estimated from the first plurality of detected landmarks, projecting landmarks from the three-dimensional model of the object into a two-dimensional space of the received image of the instance of the object using the estimated relative pose, comparing features of corresponding projected landmarks and the first plurality of detected landmarks in the two-dimensional space, and determining whether a threshold level positional correspondence exists between the positions of corresponding projected landmarks and the first plurality of detected landmarks.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to Greek Application No. 20210100909, filed December 23, 2021, and U.S. Patent Application No. 17 / 654,647, filed March 14, 2022, the entire contents of which are incorporated herein by reference.

[0002] This specification relates to the processing of objects and parts of objects, including multi-object constructs and transformed objects. [Background technology]

[0003] Image processing is a type of signal processing in which the signal being processed is an image. An input image may be processed, for example, to produce an output image or a characterization of the image.

[0004] One example of image processing is pose estimation. As described in detail below, pose estimation is a process in which the relative position and orientation of an imaging device and an object are estimated from a two-dimensional image. Pose estimation may be based on other results of image processing. One example is landmark recognition. In landmark recognition, a two-dimensional image is processed to identify landmarks and their locations in the image. The identities and locations of landmarks are examples of results on which pose estimation may be based. Summary of the Invention

[0005] In a first aspect, a method of image processing includes receiving an image of an instance of an object and a three-dimensional model of the object, detecting a first plurality of landmarks of the instance of the object in the two-dimensional image, estimating a pose of the instance of the object in the received image relative to an imaging device that captured the image, where a relative pose in the received image is estimated from the first plurality of detected landmarks, projecting landmarks from the three-dimensional model of the object into a two-dimensional space of the received image of the instance of the object using the estimated relative pose, comparing features of corresponding projected landmarks and the first plurality of detected landmarks in the two-dimensional space, and determining whether a threshold level positional correspondence exists between the positions of corresponding projected landmarks and the first plurality of detected landmarks.

[0006] Implementations of the first aspect - or the second aspect or the third aspect - may include one or more of the following features: The method may include detecting a plurality of landmarks of an instance of an object in an image, where the detected plurality includes more landmarks than a first plurality of landmarks, and selecting a first plurality of landmarks from among the landmarks of the detected plurality. The selection of the first plurality may be random. The selection of the first plurality of landmarks may be guided by a property of the landmarks. The property may be how certain it is that a given one of the landmarks has been properly detected.

[0007] The method may include, in response to determining that the threshold level positional correspondence does not exist, re-estimating a relative pose of the object instance in the first image from a second plurality of landmarks of the object instance detected in the first image, where at least some of the landmarks of the second plurality differ from the landmarks of the first plurality; projecting landmarks from the three-dimensional model of the object into a three-dimensional space of the received image of the object instance using the re-estimated relative pose; comparing features of the corresponding projected landmarks and the second plurality of detected landmarks in the three-dimensional space; and determining whether the threshold level positional correspondence exists between positions of the corresponding projected landmarks and the second plurality of detected landmarks.

[0008] The method may further include selecting a second plurality of landmarks of the object instance. The second plurality may be selected, for example, by identifying landmarks of the first plurality with a relatively large position difference between the corresponding projected landmarks and the landmarks of the first plurality, and excluding the landmarks of the first plurality with the relatively large difference from the second plurality. The second plurality may be selected, for example, by identifying the positions of the corresponding projected landmarks and the two-dimensional landmarks of the first plurality with a relatively large position difference, and excluding the landmarks of the first plurality that are in the vicinity of the corresponding projected landmarks and the detected landmarks with the relatively large position difference from the second plurality. The second plurality may be selected, for example, by identifying a misalignment direction between the corresponding projected landmarks and the detected landmarks, and excluding the landmarks of the first plurality with a misalignment direction different from the misalignment direction of the majority of the misalignments between the corresponding projected landmarks and the detected landmarks from the second plurality.

[0009] The method may include, in response to determining that a threshold level of positional correspondence exists, identifying a subset of landmarks of the object instances detected in the image with relatively large displacements from corresponding projected landmarks, and drawing a conclusion regarding the object instances based on the subset of detected landmarks. The conclusion may specify portions of the object instances that are deformed or damaged. The conclusion may quantify a magnitude of the relatively large displacement, a direction of the relatively large displacement, or both the magnitude and direction.

[0010] A threshold level of positional correspondence may be determined to exist by combining position differences between the multiple corresponding projected and detected landmarks and comparing the combination of position differences to a threshold condition. A relative pose of the object instance may be estimated by forming a first estimate of a relative pose of the object instance in the received image, identifying that the first relative pose estimate is of insufficient quality, and in response, forming a second estimate of the relative pose of the object instance in the received image.

[0011] The method may include receiving a second image of the instance of the object; detecting a second plurality of landmarks of the instance of the object in the second image acquired by a second imaging device; estimating a pose of the instance of the object in the second image relative to the second imaging device that acquired the second image, where the relative pose in the second image is estimated from the second plurality of detected landmarks; and projecting landmarks from the three-dimensional model of the object into a three-dimensional space of the second image using the estimated relative pose of the instance of the object and the estimated relative pose of the instance of the object in the second image.

[0012] The method may further include comparing features of corresponding a) the landmarks projected using the estimated relative pose of the instance of the object in the second image and b) the second plurality of detected landmarks in a two-dimensional space and determining whether a threshold level correspondence exists between the compared features of the corresponding projected landmarks and the first plurality of detected landmarks. The image of the instance may be a two-dimensional image. The detected landmarks may be two-dimensional landmarks. The received image dimensional space may be a two-dimensional space. The compared features of the corresponding projected landmarks and the second plurality of detected landmarks may be position features in the two-dimensional space.

[0013] In a second aspect, a method includes detecting landmarks of an object instance in an image of the object instance, estimating a pose of the object instance relative to an imaging device that acquired the image, projecting landmarks from a three-dimensional model of the object into a three-dimensional space of the image of the object instance using the estimated pose, identifying corresponding projected and detected landmarks, comparing features of the corresponding projected and detected landmarks in the three-dimensional space to identify a misalignment of the corresponding landmarks, identifying a subset of corresponding landmarks with a misalignment greater than a threshold condition, and identifying a deformation, movement, or blurring of the object instance or portions of the object instance based on the identified subset.

[0014] Implementations of the second aspect - or of the first aspect or the third aspect - may include one or more of the following features: Identifying the deformation, movement, or obscuration may include identifying damaged areas of the instance of the object. Identifying the deformation, movement, or obscuration may include detecting landmarks of the instance of the object in a second image of the instance of the object, and identifying the deformation, movement, or obscuration based on locations of the detected landmarks in the second image. The method may include identifying clusters of spatially close landmarks of a subset of corresponding landmarks with misalignment, and identifying the deformation or movement of the object based on the clusters of spatially close landmarks.

[0015] Identifying corresponding landmarks may include comparing contexts of the projected and detected landmarks to identify corresponding landmarks. The image of the instance may be a two-dimensional image. The detected landmarks may be two-dimensional landmarks. The dimensional space of the received image may be a two-dimensional space. The features compared to identify the misalignment may be two-dimensional position features.

[0016] In a third aspect, a method of identifying differences between two-dimensional images may include receiving a first two-dimensional image and a second two-dimensional image, each of the first image and the second image including at least a portion of a same instance of an object; receiving a three-dimensional model of the object; detecting a first plurality of two-dimensional landmarks of the instance of the object in the first image and a second plurality of two-dimensional landmarks of the instance of the object in the second image; estimating a pose of the instance of the object in each of the first image and the second image relative to one or more imaging devices that acquired the first and second two-dimensional images; projecting landmarks from the three-dimensional model of the object into two-dimensional space using at least one of the estimated relative poses; comparing positions of the detected two-dimensional landmarks to corresponding projected landmarks in each of the first and second plurality of detected two-dimensional landmarks in the two-dimensional space; and identifying an anomalous landmark among the first plurality of two-dimensional landmarks or among the second plurality of two-dimensional landmarks based on the comparison.

[0017] Other implementations of the above-mentioned methods of the first, second, and third aspects may include corresponding systems and apparatus configured to perform the actions of the methods, and computer programs tangibly embodied in machine-readable data storage devices and for configuring data processing apparatus to perform the actions.

[0018] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features and advantages of the implementations will be apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic diagram of the acquisition of a collection of different images of an object. [Diagram 2]FIG. 2 is a schematic diagram of a collection of two-dimensional images acquired by one or more cameras. [Diagram 3] FIG. 3 is a flow chart of a computer-implemented process for processing a two-dimensional image of an object. [Figure 4] FIG. 4 is a schematic diagram of an embodiment of a portion of the process of FIG. 3 performed using two different images of two different instances of the same object. [Diagram 5] FIG. 5 illustrates diagrammatically the selection of different subsets of two-dimensional landmarks during the performance of part of the process of FIG. [Figure 6] FIG. 6 is a schematic diagram of an embodiment of a portion of the process of FIG. 3 performed using three different images of four different objects. [Figure 7] FIG. 7 shows a histogram of the misalignment between landmarks detected from a 2D image and those projected from a 3D model. [Figure 8] FIG. 8 is a flowchart of a computer-implemented process for annotating landmarks that appear in a 3D model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Like reference numbers in the various drawings indicate like elements.

[0021] 1 is a schematic diagram of the acquisition of a collection of different images of an object 100. For illustrative purposes, the object 100 is shown as an assembly of idealized unmarked geometric parts (e.g., a cube, a polyhedron, a parallelepiped, etc.). However, in real applications, the object will generally have a more complex shape and will be textured or otherwise marked, for example with decorative decorations, wear marks, or other markings on the underlying shape.

[0022] A collection of one or more imaging devices (shown in this example as cameras 105, 110, 115, 120, 125) may be sequentially or simultaneously positioned at different relative positions around the object 100 and oriented at different relative angles with respect to the object 100. The positions may be distributed in three-dimensional space around the object 100. The orientation may further vary in three dimensions, i.e., all of the Euler angles (or yaw, pitch, and roll) may vary. The relative placement and orientation of the cameras 105, 110, 115, 120, 125 with respect to the object 100 may be referred to as the relative pose between the cameras and the object. Because the cameras 105, 110, 115, 120, 125 assume different relative poses, each of the cameras 105, 110, 115, 120, 125 acquires a different image of the object 100.

[0023] The relative pose between the camera and the object may be defined in a different reference coordinate system. For example, a reference coordinate system for the relative pose of the camera and the object may be defined based only on the camera and the object, e.g., by drawing a line between a point on the object and a point on the camera, and selecting a point along this line. The length of this line defines the distance between the object and the camera, and the line may be used to define the angular tilt of the camera and the object. As another example, the reference coordinate system may be defined relative to other reference points, e.g., positions on the ground or elsewhere. Distances and orientations defined relative to these points may be transformed to distances and orientations in a reference coordinate system defined only based on the camera and the object.

[0024] Returning to FIG. 2, even a simplified object such as object 100 includes many landmarks 130, 131, 132, 133, 134, 135, 136, .... Landmarks are locations of interest on object 100. Landmarks may be located at geometric locations on the object or on markings on the underlying geometry. As described in detail below, landmarks may be used to identify the pose of the object. Landmarks may also be used for other types of image processing, e.g., to classify the object, to extract features of the object, to identify the location of other structures (geometric structures or markings) on the object, to assess damage to the object, and / or to serve as origins from which measurements may be made by these and other image processing techniques.

[0025] 2 is a schematic diagram of a collection 200 of two-dimensional images acquired by one or more cameras, e.g., cameras 105, 110, 115, 120, 125 (FIG. 1). The images in collection 200 show object 100 in different relative poses. Landmarks, e.g., landmarks 130, 131, 132, 133, 134, 135, 136, ..., appear in different positions - if they appear at all - in different images. For example, in the leftmost image of collection 200, landmarks 133, 134 are obscured by the rest of object 100. In contrast, in the rightmost image 210, landmarks 131, 135, 137 are obscured by the rest of object 100.

[0026] 3 is a flow chart of a computer-implemented process 300 for processing a two-dimensional image of an object. The two-dimensional image may include multiple objects and / or a distorted arrangement of objects. Process 300 may be part of a process for identifying, for example, portions of individual objects that have been distorted, objects or portions of objects that have moved from one image to the next, and / or portions of objects that are obscured in an image. Process 300 may be performed by one or more data processing devices that perform data processing activities. The activities of process 300 may be performed according to the logic of a machine-readable instruction set, a hardware assembly, or a combination of these and / or other instructions.

[0027] At 305, a device performing process 300 receives an image of an object and a three-dimensional model of the object in the image.

[0028] The received images may be acquired by any of many different types of cameras or other imaging devices. For example, the images may be acquired by a smartphone, a digital camera, a medical imaging device, a LIDAR camera, an X-ray machine, etc. In some implementations, a single received image combines different types of information, such as information acquired by multiple imaging devices or information acquired using different imaging mechanisms. For example, a single received image may combine information acquired from different poses (stereoscopic imaging), information acquired from the same pose but using different dynamic ranges (high dynamic range imaging), information acquired using polarizing filters, etc. The received images may thus include two-dimensional information, three-dimensional information, and higher dimensional information. The information may include stereoscopic information, polarized information, high dynamic range information, depth scanning information (e.g., LIDAR), polarized and other filters, masks, labels for pixel content, vector field information of transitions (e.g., color, shape), motion information, etc. In some examples, the information is acquired using a single imaging device (e.g., a stereoscopic camera). In other examples, post-acquisition processing is performed to combine information acquired using multiple imaging devices.

[0029] In some implementations, the device performing process 300 itself acquires the received images. In other implementations, the images are received from an imaging device, either directly or through one or more intermediate devices. For example, the images may be communicated to the device performing process 300 using wired or wireless data communications, either as separate images or as part of a video stream.

[0030] A three-dimensional (3D) model generally represents an object in three-dimensional space, decoupled from any reference coordinate system. 3D models may be generated manually, algorithmically (by procedural modeling), or by scanning an actual object. Surfaces in a 3D model may be defined using texture mapping. In some instances, a 3D model of an object may be generated (e.g., using computer-aided design (CAD) software) as an assembly of constituent parts or portions of the object. For example, a 3D model of an automobile may be formed as an assembly of 3D CAD models of the constituent automobile parts, or a 3D model of a mouth may be formed as an assembly of models of the palate and crowns of the teeth in the mouth. However, in other cases, a 3D model may start as a unitary whole that is subdivided into constituent parts. For example, a 3D model of an organ may be divided into various constituent parts based on the instructions of a medical or other professional.

[0031] Although this disclosure refers to three-dimensional models "of an object" or "of the same object," a three-dimensional model is generally not a model of one physical instance of an object. Rather, a three-dimensional model is generally a general, idealized model of different objects that share common characteristics. One example includes a three-dimensional model of a particular make and model of an automobile or appliance - without considering the details of a particular instance of the automobile or appliance. Another example includes a three-dimensional model of different organs or teeth of an individual with particular physiological and / or demographic characteristics (e.g., age, sex, height, weight, jaw width, etc.).

[0032] In some examples, the image and the 3D model are of the same portion of an object. In some examples, the image includes multiple objects (or portions of multiple objects) and multiple three-dimensional models may be received. The exact nature of the object--or portion of the object--may depend on the context of the application. Exemplary objects include automobiles, internal organs, teeth, objects in a landscape (e.g., houses, streets, lampposts, rivers, etc.), etc. Considering these exemplary objects, exemplary portions of the object are: - Automotive parts (e.g. bumpers, wheels, body panels, hoods, windshields, and side panels); - parts of organs (e.g. chambers, valves, cavities, lobes, tubes, membranes, and vasculature); - Tooth parts (e.g. crown, neck, root), - parts of objects in a scene (e.g., the roof of a house, an intersection, a bend in a river), etc. Other objects and other parts of objects will be appropriate in other application contexts. Because of this variety, and for purposes of brevity, objects or parts of objects will be collectively referred to herein as "objects."

[0033] The device performing process 300 or one or more other devices may use any of a number of different approaches to ensure that the received image and the three-dimensional model are of the same object. For example, in some implementations, metadata associated with the received image may characterize the object in the image. For example, the year of make and model may be associated with a 2D image of a car. A patient name or physiological and demographic characteristics may be associated with a medical or dental image. GPS coordinates may be associated with an image of a landscape. Such metadata may be used to identify a 3D model of the same object. For example, an existing library of 3D car models may be searched for a 3D model of a car of the same make and model. Demographic and / or physiological information may be used to find a 3D model that represents a patient having those demographic and / or physiological characteristics. In some examples, a three-dimensional model may be obtained from the same instance of an object and metadata associated with the image may be used. For example, the patient name may be used to search for previously generated 3D images of the patient's physiology or to ensure correspondence between medical images acquired using one imaging modality and 3D images acquired using a different three-dimensional imaging modality.

[0034] At 310, a device performing process 300 detects landmarks in the received images of the object. The landmarks may be detected, for example, using a machine learning model. An exemplary machine learning model for landmark detection is detectron2, available at https: / / github.com / facebookresearch / detectron2. In some implementations, the landmark detection machine learning model may generate a detection score or other characterization for each landmark about how certain the landmark was properly detected.

[0035] At 315, the device performing process 300 selects a proper subset of the landmarks detected in the received image of the object. In some implementations, the selection of the landmarks is random. In other implementations, the selection of the landmarks is not random, and one or more parameters guide the selection of the landmarks. For example, in some implementations, the device performing process 300 may selectively select landmarks with particular location characteristics in the received image. For example, the device performing process 300 may selectively select landmarks that are relatively far from each other or relatively uniformly distributed across the field of view in the received image. As another example, in an implementation in which the landmark detection machine learning model generates a characterization of how certain different landmarks are properly detected, the device performing process 300 may selectively select landmarks that are detected with a relatively high degree of certainty. As another example, the selection of the landmarks may be guided by a combination of placement, certainty, and / or other parameters.

[0036] At 320, the device performing process 300 estimates the relative pose of the object in the received image using the selected landmarks. The relative pose can be estimated, for example, using a machine learning model. For example, a pose estimator that relies on landmark detection is the OpenCV function SolvePNP described in https: / / docs.opencv.org / master / d7 / d53 / tutorial_py_pose.html.

[0037] As another example, the relative pose can be estimated using forward prediction from a derived inverse model, as described in the publication entitled "iNeRF: Inverting Neural Radiance Fields for Pose Estimation" by Yen-Chen Lin et al. (arXiv:2012.05877v3, August 10, 2021), the contents of which are incorporated herein by reference (available at https: / / api.semanticscholar.org / CorpusID:228083990).

[0038] In some implementations, the quality of the pose estimation may be scored or otherwise characterized. For example, in some implementations, a binary valid / invalid characterization of the quality of the pose estimation may be generated. A "valid" pose is one that is of sufficient quality, whereas an "invalid" pose is one that is of insufficient quality. The criteria for invalidating a pose prediction may be established based on criteria that reflect real-world conditions in which the received images may be acquired. The criteria may be adjusted according to the nature of the object. For example, for a pose estimation where the object is a car, the following may be said: - The camera must be at an altitude between 0 and 5 metres relative to the ground below the car. - The camera must be within 20 meters of the vehicle. - The roll of the camera relative to the ground below the car is small (e.g., less than + / - 10 degrees), and - The boundaries of the car must for the most part match the boundaries of the car implied by the predicted pose.

[0039] If the pose estimate does not meet such criteria, the pose prediction may be marked as invalid.

[0040] As another example, in some implementations, a non-binary, more fine-grained characterization of the quality of the pose estimation may be generated. For example, additional machine learning models may be used to detect contours of the object in the received two-dimensional image. Furthermore, the estimated pose may be used to project a 3D model of the object to form a surrogate two-dimensional image of the object in the estimated pose. The detected contours from the received two-dimensional image may be compared to the contours of the object in the surrogate two-dimensional image to characterize the quality of the pose estimation. In some implementations, the results may characterize the quality of the pose estimation based on the object as a whole. For example, the correspondence between contours of the entire object in the two images may be characterized. In other implementations, the results may characterize the quality of the pose estimation based on parts or regions of the object. For example, the correspondence between contours of only parts or regions of a larger object may be characterized. It should be noted that such contour comparisons may be further used to generate a binary valid / invalid characterization of the quality of the pose estimation.

[0041] As yet another example, in some implementations, the quality of the pose estimation may be estimated from the context around each landmark. For example, the context of a detected landmark in a received image may be compared to the context of a possible corresponding landmark in a proxy image formed from a 3D model. For example, 3D landmarks of an object from a 3D model may be projected onto a two-dimensional or three-dimensional proxy image in an estimated pose. Such landmark context may include, for example, shape and / or visual properties of features around the landmark, such as, for example, color, structure, pattern, optical properties (e.g., reflectance, polarization), size of typical adjacent structures, etc. In some implementations, these properties are learned from example images or computed from a 3D model (e.g., door handles are typically attached to doors, and doors have a particular separation from other parts of a car, etc.).

[0042] As yet another example, in some implementations, the quality of the pose estimation may be characterized by estimating the pose multiple times and comparing the different pose estimates. For example, different true subsets of landmarks detected in the received images of the object may be selected, for example, at 315. The relative pose of the object in the received images may be estimated multiple times using different true subsets to determine the stability of the different pose estimates, i.e., how far the different pose estimates deviate from each other. In some implementations, for example, the stability is evaluated using landmarks from portions of the received images rather than the entire received image. For example, the portions may be defined to exclude landmarks in the object of interest or other highly variable portions of the received images. In such cases, the landmarks used to estimate the quality of the pose estimation are separated from other more variable landmarks. If the quality of the pose estimation is insufficient, the pose may be re-estimated until the quality is sufficient. For example, the process 300 may return to 315 to select a different subset of landmarks and a re-estimated pose using the different subsets. In some examples, process 300 may return to 310 to detect additional landmarks or to detect landmarks using a different approach, such as a different or fine-tuned machine learning model. If the quality of the pose estimation remains insufficient, process 300 may be stopped for a given received image and a different image may be received and used.

[0043] At 325, the device performing process 300 uses the three-dimensional model of the object received at 305 and the relative pose estimated at 320 to project landmarks from the three-dimensional model into the same dimensional space as the received image. Essentially, the operations performed by the device performing process 300 orient and position the three-dimensional model to match the relative pose estimated at 320. Three-dimensional landmarks in the three-dimensional model that are visible in the virtual received image formed in the same dimensional space as the received image may be identified.

[0044] After projection, corresponding projected and detected landmarks may be identified. For example, the context of the projected and detected landmarks may be compared to identify corresponding landmarks. Such landmark context may include, for example, shape and / or visual characteristics of features around the landmark, such as, for example, color, structure, pattern, optical properties (e.g., reflectance, polarization), size of typical adjacent structures, etc. In some implementations, these characteristics are learned from example images or computed from a 3D model (e.g., door handles are typically attached to doors, and doors have a particular separation from other parts of an automobile, etc.).

[0045] At 330, the device performing process 300 compares the positions or other characteristics of the landmarks projected onto the virtual image at 325 with the positions or other characteristics of corresponding landmarks detected in the image received at 310. The comparison is performed for each landmark in at least some of the dimensions into which the three-dimensional model was projected at 325.

[0046] The results of the comparison may be expressed in a variety of different ways depending on the particular characteristics being compared. For example, assume that the 2D or 3D position of each of the projected individual landmarks at 325 is compared to the 2D or 3D position of the corresponding individual landmark detected at 310. The 2D or 3D position difference may be expressed in terms of both magnitude and direction. The magnitude of the separation between the landmarks may be expressed, for example, in units of pixels as a percentage of the width of the received image, or otherwise. As a further example, color differences may be expressed in wave numbers, reflectance differences may be expressed in radiometric units, and polarization differences may be expressed as angle differences.

[0047] In some implementations, a device performing process 300 may further generate a value that characterizes a combination of differences of several corresponding landmarks. For example, an average difference in 2D or 3D positions for a set of corresponding landmarks, or an algebraic or vector sum of the differences in 2D or 3D positions, may be generated and used in subsequent activities.

[0048] At 335, the device performing the process 300 determines whether a threshold level correspondence exists between the landmark features projected onto the virtual image at 325 and the landmark features detected in the image received at 310. Both the threshold level--and the feature differences compared to the threshold level--can be landmark-by-landmark or combination-by-combination. For example, the number or percentage of corresponding landmarks with individual differences below a threshold condition can be used to determine whether a threshold level correspondence exists. As another example, a vector sum of 2D or 3D position differences of several corresponding landmarks can be compared to a threshold level correspondence to determine whether a threshold level correspondence exists.

[0049] In either case, the threshold level may be expressed in objective terms that are independent of the particular instance of process 300, or in subjective terms that are tailored to the particular instance of process 300. For example, in some implementations, an objective threshold level, such as, for example, a particular number of pixels or a percentage of the width of the received image, may be applied to multiple instances of process 300. In other implementations, a subjective threshold level, such as, for example, a standard deviation of 2D or 3D position differences, or a value tailored to the certainty that landmarks have been properly detected in the received image, may be applied during different instances of process 300.

[0050] In response to determining that the threshold level correspondence does not exist, the device performing process 300 selects at 340 a different real subset of the landmarks detected in the received image (i.e., a different real subset of the landmarks detected at 310). In some implementations, the landmarks are selected randomly. In other implementations, the selection of the landmarks is not random, and the selection of the landmarks is guided. In addition to the parameters described above at 315, the results of the landmark-by-landmark comparison at 330 may further guide the selection of the different subset of landmarks. For example, in some implementations, if the difference between the landmarks projected at 325 and the corresponding individual landmarks detected at 310 is relatively small, that individual landmark may be selectively included in the different real subset. As another example, in some implementations, if the difference between the landmarks projected at 325 and the corresponding individual landmarks detected at 310 is relatively large, that individual landmark may be excluded from the different real subset. For example, with reference to histogram 700 (FIG. 7), landmarks from cluster 720 of bars may be selected for the distinct real subsets, while landmarks from bar 715 may be excluded from the distinct real subsets.

[0051] After selecting a different true subset at 340, the device performing process 300 estimates a new relative pose of the object using the different true subset at 320, projects landmarks from the 3D model into the three-dimensional space of the received image using the new relative pose at 325, and compares the positions of the landmarks at 330. This may be repeated until a threshold level of correspondence is determined to exist at 335.

[0052] In response to determining that a threshold level of correspondence exists, the device performing process 300 identifies anomalous landmarks among the landmarks detected in the received image at 345. The anomalous landmarks may be detected among all of the landmarks detected at 310, among the landmarks in the subset that provides the threshold level of correspondence at 335, or among a different subset of the landmarks detected at 310.

[0053] Anomalous landmarks may be detected in many different ways. For example, in some implementations, a histogram of the differences between corresponding landmarks (e.g., histogram 700, FIG. 7) may be generated and used to identify anomalies. As another example, a threshold difference may be used to identify anomalies. In some implementations, the threshold difference may be expressed in objective terms (e.g., above a certain number of pixels or percentage of the width of the received image). In some implementations, the threshold position difference may be expressed in subjective terms, such as in terms relative to a particular instance of process 300. For example, a standard deviation from a mean difference, or a direction relative to the difference in 2D or 3D orientation of other corresponding landmarks in a particular instance of process 300, may be used to identify anomalies.

[0054] The identified anomaly landmarks may be applied to a variety of different activities depending on the operational context. For example, in an operational context where damage or deformation of an object instance is identified, the anomaly landmarks may be used to identify the damaged or deformed portions of the object instance. For example, a cluster of spatially close anomalies may indicate that an underlying portion of the object instance is damaged or deformed in the vicinity. As another example, the anomaly landmarks may be used to characterize the extent of the damage or deformation. For example, the magnitude of the position difference - similar to color or other optical differences - may be taken as an indication of the extent of the damage or deformation. If the difference is relatively small, this may be interpreted as an indication of normal wear and damage to the object instance. On the other hand, if the difference is relatively large, this may be interpreted as an indication of more severe damage to the object instance.

[0055] As another example, in a motion context in which motion of a portion of an object instance is specified, the anomaly landmarks may be used to characterize the motion. For example, the magnitude and direction of the position difference may be taken as an indication of the magnitude and direction of the motion. Examples of such motion contexts include contexts in which motion of, for example, a moving arm or other joint of a robot or other automated machine part is specified.

[0056] As yet another example, in an operational context where obscuration of portions of an object instance is identified, anomalous landmarks may be used to characterize the obscured portions. For example, color, or other optical differences may be used if the obscuration is a decorative decoration, or even a new paint coating. When 2D or 3D position differences are used to identify obscuration, the anomalous landmarks may be anomalous in the sense that they are not detected in the received images. In some implementations, the positions of the anomalous landmarks may be used to define a rough boundary of the obscured body. In some implementations, multiple executions of process 300 using different images may be used to characterize the movement of the obscured body between images, as different landmarks will be anomalous to be not detected.

[0057] As another example, in some motion contexts, soft body deformations are identified. In such contexts, anomaly landmarks can be used to characterize the deformation of an object using a "wireframe" 3D model, e.g., to establish the kinematic placement of the wireframe using the anomaly landmarks.

[0058] As another example, in some operational contexts, growth of an object is identified. In such contexts, the nature of the abnormal landmarks may be used to identify the type of growth. For example, if the volume of an object is growing in three dimensions, the 2D position difference of the corresponding two-dimensional landmarks may reflect their distance from a reference position on the object. As another example, if the object is growing (e.g., lengthening) in only one dimension, the 2D position difference may reflect the distance of the object from a reference line or plane. In such cases, the "abnormal" landmarks may include a relatively large percentage - or even a majority - of the landmarks.

[0059] 4 is a schematic diagram of an exemplary implementation of a portion of process 300 performed using two different two-dimensional images of two different instances of the same object. As noted above, the images received at 305 may contain information in multi-dimensional space, and comparisons may be made in such dimensions. However, for purposes of illustration, an exemplary implementation of a portion of process 300 is performed using a first two-dimensional image 405 and a second two-dimensional image 410. Moreover, all comparisons are in two-dimensional space.

[0060] Image 405 is a 2D image of an instance of an object 415 that has been deformed to a relatively small extent. Image 410 is a 2D image of an instance 420 of the same object that has been deformed to a relatively larger extent. Both object instances 415, 420 are represented by the same 3D model 425. In particular, 3D model 425 is a generic, idealized representation of object instances 415, 420 in three-dimensional space. For example, 3D model 425 can be a CAD model or a process model of the object, e.g., without any deformations or obscurations.

[0061] As described above, the 2D images 405, 410 and the 3D model 425 are received at 305 (FIG. 3) by one or more data processing devices performing data processing activities. In a schematic diagram, a collection of three-dimensional landmarks 430 is shown in the 3D model 425 as an arrangement of unfilled points at different locations in the 3D model 425. The three-dimensional landmarks 430 are points of interest in the 3D model 425. In some implementations, the three-dimensional landmarks 430 are identified in metadata that accompanies the 3D model 425 when it is received at 305 (FIG. 3). In other implementations, a computer-implemented process may be used to annotate the landmarks to the 3D model 425. An example of such a process is process 800 (FIG. 8) described below.

[0062] The receiving device detects at 310 (FIG. 3) a collection of two-dimensional landmarks 435 in the image 405, 410. For illustrative purposes, the 2D landmarks 435 are shown as a two-dimensional arrangement of solid black dots at different positions along the dashed outline of the object instance 415, 420. In a real implementation, such an arrangement is unnecessary and the positions of the 2D landmarks 435 may be noted using 2D position coordinates or otherwise. In the schematic diagram shown, the 2D landmarks 435 are corners or other edge features in the object instance 415, 420. This is not necessarily the case. The 2D landmarks 435 may be located at other times in the object instance 415, 420. For example, the 2D landmarks 435 may be located at the connections between different components, at decorative features in the surface of the object instance 415, 420, or at other times in the object instance 415, 420.

[0063] The receiving device further selects 315 (FIG. 3) a subset of the 2D landmarks 435 in each collection of 2D landmarks 435. For purposes of illustration, the subset of 2D landmarks 435 is selected from a similar contiguous area 440 in both images 405, 410. However, this is not necessarily the case. The selected 2D landmarks 435 may be distributed - randomly or otherwise - across the images 405, 410 such that there is not an "area" per se in which landmarks are selected.

[0064] Using the selected subset of 2D landmarks 435, the device estimates 320 (FIG. 3) the relative pose of the object instances 415, 420 in the images 405, 410. For the selected subset of 2D landmarks 435 from the image 405, they are observed in portions of the object instance 415 that are not deformed or that are deformed to a relatively small extent. For example, the 2D landmark 450 is observed near portions of the object instance 415 that are deformed to a relatively small extent. In contrast, for the selected subset of 2D landmarks 435 from the image 410, the 2D landmarks 435 are observed in portions of the object instance 420 that are deformed to a relatively large extent. For example, the 2D landmarks 455, 460, 465 are observed near portions of the object instance 415 that are deformed to a relatively large extent.

[0065] The estimated pose of object instance 415 is relatively accurate because the 2D landmarks 435 in the selected subset of images 405 are where they are supposed to be. Any error in the estimation due to 2D landmarks 450 is relatively small. In contrast, the estimated pose of object instance 420 is relatively inaccurate because the 2D landmarks 435 in the selected subset of images 410 are significantly displaced from where they are supposed to be. Indeed, in some instances, the pose estimation may return unacceptably inaccurate results, or even no results, and the pose may be re-estimated, for example, using a different subset of detected landmarks or landmarks detected using a different or fine-tuned machine learning model.

[0066] Using the estimated relative pose, the device projects 3D landmarks from the 3D model 425 onto a virtual two-dimensional image at 325 (FIG. 3). For the pose estimated from the image 405, this obtains a collection of landmarks 470. The landmarks in collection 470 are "two-dimensional" in that they are located only in two-dimensional space (as opposed to the landmarks 430 in three-dimensional space in the 3D model 425). Although the landmarks in collection 470 are two-dimensional, they are further shown in collection 470 as unfilled dots to show their correspondence to the 3D landmarks 430 in the 3D model 425.

[0067] The device compares 330 (FIG. 3) the positions of the 2D landmarks 435 detected in the image 405 with the positions of the 2D landmarks in the collection 470. This comparison is represented diagrammatically in the lower left corner of FIG. 4. Where a 2D landmark in the collection 470 overlaps or nearly overlaps with a 2D landmark 435, an unfilled dot is superimposed with an "x". However, where the 2D landmark 435 and the 2D landmark in the collection 470 are drawably offset, the 2D landmark 435 is represented by a filled black dot. In the embodiment shown, two different 2D landmarks 435 - namely, 2D landmark 450 and 2D landmark 475 - are drawably offset. The corresponding 2D landmark in the collection 470 is represented by an unfilled dot without an "x".

[0068] Based on a comparison of the locations of the 2D landmarks 435 detected in the image 405 to the locations of the 2D landmarks in the collection 470, the device determines whether a threshold level of correspondence has been reached, at 335 (FIG. 3). Additionally, the device identifies anomalous landmarks (e.g., 2D landmarks 450, 475), at 345 (FIG. 3).

[0069] In the illustrated embodiment, the landmark collection formed by projecting the estimated pose from the image 410 is not shown. This is only a schematic diagram. A landmark collection may be formed and the positions of the 2D landmarks 435 detected in the image 410 may be compared with the positions of the 2D landmarks in such collection. However, such a comparison results in a larger deviation between the positions of the landmarks. Indeed, in some implementations, it may be difficult to identify the corresponding landmarks.

[0070] When faced with a misregistration greater than the threshold level of correspondence or other indications of inaccuracy, the device selects a different subset of landmarks 435 from the image 410 at 335 (FIG. 3). Again, the relative pose of the object 420 may be estimated using the selected subset at 320 (FIG. 3), 3D landmarks may be projected from the 3D model 425 onto the two-dimensional image using the estimated relative pose at 325 (FIG. 3), and the positions of the 2D landmarks are compared to the positions of the 2D landmarks formed by projecting the 3D landmarks at 330 (FIG. 3). This process may be repeated until the misregistration is no longer greater than the threshold level of correspondence.

[0071] 5 illustrates diagrammatically the selection of different subsets of 2D landmarks 435 detected in an image 410 that may prove suitable for accurately estimating the relative pose of an object instance 420. In particular, the selected landmarks are located in an area 505 that encompasses parts of the object instance 420 that are undeformed or that are deformed only to a relatively small extent. Because the 2D landmarks 435 in the area 505 are located where they are expected to be, the estimated pose of the object instance 420 may be relatively accurate.

[0072] As previously mentioned, at 330 (FIG. 3), the positions of the 2D landmarks 435 detected in the image 410 may be compared to the positions of the 2D landmarks projected from the 3D model 425. This comparison is represented diagrammatically at the bottom of FIG. 5, where the superimposed projected and detected ones are shown using unfilled dots superimposed with an "x". The remainder of the detected 2D landmarks 435 are represented by filled black dots. The remainder of the projected 3D landmarks are represented by unfilled dots.

[0073] 6 is a schematic diagram of some implementations of process 300 performed using three different images 605, 610, 615 of four different objects 620, 625, 630, 635. Again, images 605, 610, 615 are shown as two-dimensional for illustrative purposes. Images 605, 610, 615 differ from each other in that a pose 620, 625, 630, 635 of at least one object differs in the images. In the implementation shown, the pose of object 625 differs in images 605, 610, 615. In other implementations, the pose of one or more other objects 620, 630, 635 may differ - whether or not the pose of object 625 also differs.

[0074] In some implementations, the device performing process 300 may receive an independent 3D model of each of the objects 620, 625, 630, 635. In other implementations, the device performing process 300 may receive only a single 3D model of one of the objects 620, 625, 630, 635. For example, the device may receive only the 3D model 425 (FIG. 4) of the object 620.

[0075] The receiving device detects 310 (FIG. 3) a collection of two-dimensional landmarks 640, 645, 650 in the images 605, 610, 615, 410. For illustrative purposes, the 2D landmarks 640, 645, 650 are again shown as two-dimensional arrangements of solid black dots at different locations along the dashed outlines of the object instances 620, 625, 630, 635.

[0076] The receiving device further selects 315 (FIG. 3) a subset 660, 665, 670 of 2D landmarks in each collection of 2D landmarks 640, 645, 650. For illustrative purposes, the subsets 660, 665, 670 of 2D landmarks are selected from similar areas 655 in the images 605, 610, 615. However, the selected 2D landmarks may also be distributed across the images 605, 610, 615.

[0077] Using the selected subset 660, 665, 670 of the 2D landmarks 640, 645, 650 and one or more corresponding 3D models, the device estimates 320 (FIG. 3) the relative pose of at least one object instance 620, 625, 630, 635 in the images 605, 610, 615.

[0078] In some examples, the subsets 660, 665, 670 of landmarks are from at least one of the objects 620, 625, 630, 635, and the estimated pose of the objects 620, 625, 630, 635 is relatively accurate. For example, referring to the subsets 660, 670, a relatively large number of landmarks are selected from the object 635. Assuming a 3D model of the object 635 was available, the relative pose of the object 635 could be estimated relatively accurately.

[0079] In other cases, the estimated poses of the objects 620, 625, 630, 635 are relatively inaccurate. For example, - too few landmarks were selected from the objects 620, 625, 630, 635 for which a corresponding 3D model exists; - the part of the object 620, 625, 630, 635 for which there is a corresponding 3D mode is obscured, or - Difficulties in assigning landmarks to the corresponding 3D model (e.g. due to damage or deformation); There are several possible contributions to this imprecision, including:

[0080] For example, with reference to subset 645, even if a relatively large number of landmarks are selected from object 620, a pose of object 625 that obscures portions of object 620 may cause pose estimation based on object 620 to be relatively inaccurate. Indeed, in some instances, pose estimation may return unacceptably inaccurate results, or even no results.

[0081] Using each estimated relative pose, the device projects 3D landmarks from one or more 3D models onto the two-dimensional image to obtain a respective collection of landmarks at 325 (FIG. 3). The device further compares the positions of the 2D landmarks 640, 645, 650 detected in the images 605, 610, 615 with the positions of the 2D landmarks projected from the 3D models at 330 (FIG. 3).

[0082] At this point, assuming a threshold level of correspondence has been reached, the projection of the 3D landmarks from the 3D model onto the two-dimensional image at 325 may be used to identify anomalous landmarks by a variety of different techniques. For example, with reference to image 610, a comparison of the locations of 2D landmarks 645 with the locations of 2D landmarks projected from a 3D model of object 620 (e.g., 3D model 525, FIG. 4) shows that many landmarks from object 620 were not detected in image 610. If no landmarks from object 620 are detected at all, this may be interpreted as an indication that object 620 is partially obscured in image 610. Furthermore, a rough outline of the obscured portion may be formed based on the locations of the undetected landmarks.

[0083] As another example, the positions of 2D landmarks 640, 645, 650 identified in images 605, 610, 615 may be compared to one another to, for example, identify that the relative pose of the objects has changed. For example, with reference to images 605, 615, a comparison of the positions of the 2D landmarks 640, 650 indicates that the positions of the 2D landmarks from objects 620, 630, 635 have not substantially changed. In contrast, the positions of the 2D landmarks from object 625 have not substantially changed, but indicate that the pose of object 625 in images 605, 615 is different.

[0084] 7 is a histogram 700 illustrating the misalignment between landmarks detected from a received image and landmarks projected from a 3D model. Although devices implementing the methods described herein generally do not form and display histograms such as histogram 700 itself, histogram 700 illustrates how misalignment between corresponding landmarks may come into play in various activities of these methods. As discussed above, the misalignment may be 2D misalignment, 3D misalignment, or even misalignment in different dimensions (such as color, reflectance, polarization, etc.).

[0085] The histogram 700 includes a horizontal axis 705 and a vertical axis 710. The horizontal axis 705 is divided into a number of intervals, each of which encompasses a range of displacements between corresponding detected and projected landmarks. For example, one such interval encompasses zero displacements between corresponding landmarks - as would be the case if the corresponding landmarks were identical. The position along the vertical axis 710 represents the number of corresponding landmarks with displacements in each interval. A bar that extends far up from the horizontal axis 705 indicates that the number of corresponding landmarks across that bar within the displacement range of the horizontal axis 705 is greater than the number in bars that do not extend as far from the horizontal axis 705. For example, for most of the displacement range of the horizontal axis 705, the number of corresponding landmarks appears to be zero. However, a discernible number of corresponding landmarks with displacements within the range encompassed by the bar 715. Furthermore, a relatively large number of corresponding landmarks with displacements within the range encompassed by the bars in the cluster 720.

[0086] As discussed above, histogram 700 may show how position and other misalignments between corresponding landmarks may play a role in various activities in these methods. For example, assume histogram 700 represents misalignments of a first subset of corresponding landmarks, such as those resulting from a comparison at 330 (FIG. 3). Corresponding landmarks with misalignments within the range encompassed by bar 715 may cause the average correspondence to drop below a threshold level. For example, this may be due to a relatively large misalignment of the corresponding landmarks themselves, but also of the landmarks detected from the received image, making the relative pose estimated at 325 (FIG. 3) less accurate.

[0087] In this situation, the guided selection of different subsets of detected landmarks at 340 (FIG. 3) may exclude landmarks with displacements from the projected landmarks within the range encompassed by bar 715. Additionally, detected landmarks with displacements within the range encompassed by the bars in cluster 720 may be selectively selected. In some situations, other detected landmarks (i.e., landmarks with displacements that do not appear in histogram 700) may also be selected.

[0088] Such guided selection may be repeated multiple times, with each iteration increasing the accuracy of the relative pose estimation. For example, the bars in cluster 720 may appear to be close together on the horizontal axis 705 of the scale of presence, but a change in that scale may indicate that other landmarks with misalignments within the range encompassed by the bars in cluster 720 should be excluded from the next subset.

[0089] As another example of how misalignment between corresponding landmarks may function in the methods described herein, assume that histogram 700 represents all misalignments of corresponding landmarks, such as may be identified after reaching a threshold level of correspondence at 335 (FIG. 3). In this case, corresponding landmarks with misalignments within the range encompassed by bar 715 may be identified as anomalous and may serve as a basis for drawing conclusions regarding object instances in images in which the landmarks are detected. For example, detected landmarks with 2D or 3D misalignments within the range encompassed by bar 715 may be identified from portions of object instances that are deformed or damaged. As another example, detected landmarks with 2D or 3D misalignments within the range encompassed by bar 715 may be identified from portions of object instances that have moved between images. As yet another example, detected landmarks with misalignments in color or other optical properties within the range encompassed by bar 715 may be from objects or portions of objects that are obscured in the images, such as by decorative decorations or paint coatings.

[0090] 8 is a flow chart of a computer-implemented process 800 for annotating landmarks that appear in a 3D model, such as, for example, 3D model 425. Process 800 may be performed, for example, by one or more data processing devices that perform data processing activities according to the logic of a set of machine-readable instructions, a hardware assembly, or a combination of these and / or other instructions. Process 800 may be performed independently or in combination with other activities. For example, process 800 may be performed in combination with process 300 (FIG. 3).

[0091] At 805, the system performing process 800 renders a collection of proxy images of the object using the 3D model of the object formed from its constituent parts. The proxy images are not actual images of the real-world object. Rather, the proxy images are substitutes for images of the real-world object. These proxy images show the object from a variety of different angles, orientations, and / or dimensions, generally with the same dimensionality as the images received at 305 in process 300 (FIG. 3) - as if a camera were imaging the object in a variety of different relative poses.

[0092] The surrogate image may be rendered using the 3D model in many ways. For example, ray tracing or other computer graphic techniques may be used. Typically, a 3D model of an object is perturbed to render the surrogate image. Thus, different surrogate images may show different variations of the 3D model. Typically, the perturbation may mimic real-world variations in the object--or parts of the object--represented by the 3D model. For example, in a 3D model of an automobile, the color of the exterior paint and interior trim may be varied. In some examples, parts (features such as tires, hubcaps, and roof racks) may be added, removed, or replaced. As another example, in a 3D model of an organ, physiologically relevant size and relative size variations may be used to perturb the 3D model.

[0093] In some implementations, aspects other than the 3D model may be varied to further alter the proxy image. - variations in imaging devices (e.g. camera resolution, zoom, focus, aperture speed), - variations in image processing (e.g. digital data compression, chroma subsampling), and - Variations in imaging conditions (e.g. lighting, weather, background color, and shape) The present invention may mimic real-world variations, including:

[0094] In some implementations, the surrogate image is rendered in a reference coordinate system. The reference coordinate system may include background features that appear behind the object, and foreground features that appear in front of the object - and potentially obscuring parts of the object. Generally, the reference coordinate system reflects the real-world environment in which the object may be observed. For example, a car may be rendered in a reference coordinate system that resembles a parking lot, while an organ may be rendered in a physiologically relevant context. The reference coordinate system may be further altered to further alter the two-dimensional image.

[0095] In general, it is desirable for the proxy images to vary widely. Furthermore, the number of proxy images - and the degree of variation - may depend on the complexity of the object and the image processing that is ultimately performed using the landmarks annotated in the 3D model. As an example, over 2000 highly diverse proxy images (with relative poses and displacements) of a car may be rendered. Because the proxy images are rendered from the 3D model, complete knowledge information about the object's position in the proxy images may be preserved regardless of the number of proxy images and the degree of variation.

[0096] At 810, the system performing process 800 assigns each region of the object shown in the substitute image to a part of the object. As described above, the 3D model of the object may be divided into identifiable constituent parts based on function and / or structure. When the substitute image of the 3D model is rendered, the part to which each region in the image belongs may be preserved. Regions - which may be pixels or other areas in a two-dimensional image - may thus be assigned to corresponding constituent parts of the 3D model using complete knowledge information derived from the 3D model.

[0097] At 815, the system performing process 800 identifies identifiable regions of the part in the substitute image. Identifiable regions of the part are areas (e.g., pixels or groups of pixels) that can be identified in the substitute image using one or more image processing techniques. For example, in some implementations, corners of regions in each image assigned to the same part are detected using, for example, a Moravec corner detector or a Harris corner detector (https: / / en.wikipedia.org / wiki / Harris_Corner_Detector). As another example, an image feature detection algorithm, for example SIFT / SURF / HOG / (https: / / en.wikipedia.org / wiki / Scale-invariant_feature_transform), can be used to define identifiable regions.

[0098] At 820, the system performing process 800 identifies a collection of landmarks in the 3D model by projecting identifiable regions in the proxy image back onto the 3D model. Volumes in the 3D model that correspond to the identifiable regions in the proxy image are identified as landmarks in the 3D model.

[0099] In some implementations, one or more filtering techniques may be applied before or after backprojection onto the 3D model to reduce the number of these landmarks and ensure quality. For example, in some implementations, regions in the proxy image that are close to the outer boundary of the object may be discarded before backprojection. As another example, backprojection of regions that are too far away from the corresponding part in the 3D model may be discarded.

[0100] In some implementations, only volumes in the 3D model that meet the threshold criteria are identified as landmarks. The threshold criteria can be identified in many ways. For example, volumes that are candidate landmarks in the 3D model and identified by backprojection from different surrogate images rendered using different relative poses and variations can be collected. Clusters of candidate landmarks can be identified and anomalous candidate landmarks can be discarded. For example, a clustering technique, such as the OPTICS algorithm (https: / / en.wikipedia.org / wiki / OPTICS_algorithm, a variation of DBSCAN, https: / / en.wikipedia.org / wiki / DBSCAN), can be used to identify clusters of candidate landmarks. The effectiveness of the clustering can be evaluated using, for example, the Calinski-Harabasz index (i.e., a variance ratio criterion) or other criteria. In some implementations, the clustering technique can be selected and / or adjusted (e.g., by tuning hyperparameters of the clustering algorithm) to improve the effectiveness of the clustering. If necessary, candidate landmarks that are in a cluster and are closer to each other than the threshold criteria can be merged. In some implementations, candidate landmark clusters in different parts of the 3D model may be further fused into one cluster, and in some implementations, the centroids of several candidate landmarks in a cluster may be represented as one landmark.

[0101] In some implementations, landmarks in the 3D model may be filtered based on the accuracy with which their positions or other characteristics can be predicted in a proxy image rendered from the 3D model. For example, if a 3D landmark's position in the proxy image is too difficult to predict (e.g., predicted inaccurately more than a threshold percentage of the time or predicted with insufficient accuracy), the 3D landmark may be discarded. As a result, only 3D landmarks with positions in the proxy image that the landmark predictor can predict relatively easily remain.

[0102] In some examples, the number of identified landmarks may be tailored to a particular data processing activity. - Rendering more or less substitute images, particularly using more or less permutations of the 3D model, in 805; - dividing the 3D model into more or fewer parts with assigned areas; - relaxing or tightening the constraints for considering an area as identifiable in 815, and / or - After 820, relax or tighten the constraints for filtering landmarks after backprojecting identifiable regions onto the 3D model. The amount of the ion exchange reaction can be adjusted in a number of ways, including by

[0103] The subject matter and embodiments of the operations described herein may be implemented in digital electronic circuitry, including the structures disclosed herein and structural equivalents thereof, or in computer software, firmware, or hardware, or in any combination of one or more of them. The subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded in a computer storage medium for execution by or to control the operation of a data processing apparatus. Alternatively, or in addition, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to an appropriate receiver device for execution by the data processing apparatus. The computer storage medium may be or be included in a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Furthermore, although a computer storage medium is not a propagated signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be, or be contained in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0104] The operations described herein may be realized as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or received from other sources.

[0105] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a plurality or combination of these. The apparatus may encompass special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). In addition to hardware, the apparatus may further include code that creates an execution environment for the computer program in question, such as code constituting a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform execution environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0106] A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may correspond to a file in a file system, but this is not required. A program may be stored in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in one file assigned to the program in question, or in multiple coordinated files (e.g., a file storing one or more modules, subprograms, or code portions). A computer program may be deployed to be executed in one computer, or in multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0107] The processes and logic flows described herein may be implemented by one or more programmable processors executing one or more computer programs to perform operations by processing input data and generating output. The processes and logic flows may further be implemented by, and an apparatus may be realized as, special purpose logic circuitry, such as, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0108] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for performing operations according to the instructions, and one or more memory devices for storing instructions and data. Generally, a computer further includes one or more mass storage devices for storing data, such as, for example, a magnetic disk, a magneto-optical disk, or an optical disk, or is operatively coupled to receive data from, or transmit data to, the one or more mass storage devices, or both. However, a computer need not have such devices. Furthermore, a computer may be incorporated in another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0109] To provide for interaction with a user, embodiments of the subject matter described herein may be implemented in a computer that includes a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to a user, and a keyboard and a pointing device, such as a mouse or trackball, by which the user may provide input to the computer. Other types of devices may be used to provide for interaction with a user as well, for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. In addition, the computer may interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser at the user's client device in response to a request received from the web browser.

[0110] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some instances, the actions recited in the claims may be performed in a different order and still achieve desirable results. In addition, the steps depicted in the accompanying figures do not necessarily require the particular order or sequence shown to achieve desirable results. In certain implementations, multitasking and parallel processing may be beneficial.

[0111] A number of embodiments are described. Nevertheless, it is understood that various modifications may be made and, therefore, other embodiments are within the scope of the following claims.

Claims

1. 1. A method of image processing, said method comprising: receiving an image of an instance of an object and a three-dimensional model of the object; Detecting a first plurality of landmarks of the instance of the object in a two-dimensional image; estimating a pose of the instance of the object in the received image relative to an imaging device that captured the image, where the relative pose in the received image is estimated from the first plurality of detected landmarks; using the estimated relative pose to project landmarks from the three-dimensional model of the object into the three-dimensional space of the received image of the instance of the object; comparing features of the corresponding projected landmarks and the first plurality of detected landmarks in the dimensional space; determining whether a threshold level positional correspondence exists between corresponding projected landmarks and the first plurality of detected landmarks and locations; A method comprising:

2. The method further comprising: detecting a plurality of landmarks of the instance of the object in the image, the detected plurality including more landmarks than the first plurality of landmarks; selecting the first plurality of landmarks from among the detected plurality of landmarks; The method of claim 1 further comprising:

3. the selection of the first plurality is random; The method of claim 2.

4. Selection of the first plurality of landmarks is guided by characteristics of the landmarks. The method of claim 2.

5. the characteristic being how certain a given one of the landmarks was properly detected; The method according to claim 4.

6. In response to determining that the threshold level of positional correspondence does not exist, re-estimating the relative pose of the instance of the object in the first image from a second plurality of landmarks of the instance of the object detected in the first image, where at least some of the landmarks in the second plurality are different from the landmarks in the first plurality; re-estimating the relative pose and projecting landmarks from the three-dimensional model of the object into the three-dimensional space of the received image of the instance of the object; comparing features of the corresponding projected landmarks and the second plurality of detected landmarks in the dimensional space; determining whether the threshold level of positional correspondence exists between positions of corresponding projected landmarks and the second plurality of detected landmarks; The method of claim 1 further comprising:

7. and selecting a second plurality of landmarks of the instance of the object, wherein selecting the second plurality of landmarks comprises: identifying corresponding projected landmarks and the landmarks of the first plurality with a relatively large position difference; removing the landmarks in the first plurality with a relatively large difference from the second plurality; The method of claim 6, comprising:

8. and selecting a second plurality of landmarks of the instance of the object, wherein selecting the second plurality of landmarks comprises: identifying corresponding projected landmarks and two-dimensional landmarks of the first plurality with a relatively large position difference; removing landmarks from the second plurality that are in the vicinity of the corresponding projected landmark and the detected landmark with the relatively large position difference; The method of claim 6, comprising:

9. and selecting a second plurality of landmarks of the instance of the object, wherein selecting the second plurality of landmarks comprises: determining a misalignment direction between the corresponding projected landmark and the detected landmark; removing the landmarks of the first plurality from the second plurality with a misalignment direction different from the misalignment direction of a majority of the misalignments between the corresponding projected landmarks and the detected landmarks; The method of claim 6, comprising:

10. In response to determining that the threshold level of positional correspondence exists, identifying a subset of the landmarks of the instances of the object detected in the image that have a relatively large deviation from the corresponding projected landmarks; drawing a conclusion about the instance of the object based on the subset of the detected landmarks; The method of claim 1 further comprising:

11. and drawing said conclusion includes designating portions of said instance of said object that are deformed or damaged. The method of claim 10.

12. and wherein said drawing a conclusion includes quantifying a magnitude of the relatively large misalignment, a direction of the relatively large misalignment, or both the magnitude and the direction. The method of claim 10.

13. Determining whether the threshold level of positional correspondence exists includes: combining position differences between a plurality of corresponding projected landmarks and the detected landmarks; comparing the combination of position differences with a threshold condition; The method of claim 1 , comprising:

14. estimating the relative pose of the instance of the object, forming a first estimate of the relative pose of the instance of the object in the received image; Identifying a first relative pose estimate of insufficient quality; in response, forming a second estimate of the relative pose of the instance of the object in the received image; and The method of claim 1 , comprising:

15. receiving a second image of the instance of the object; detecting a second plurality of landmarks of the instance of the object in the second image acquired by a second imaging device; estimating a pose of the instance of the object in the second image relative to the second imaging device that acquired the second image, the relative pose in the second image being estimated from the second plurality of detected landmarks; projecting the landmarks from the three-dimensional model of the object into the three-dimensional space of the second image using the estimated relative pose of the instance of the object and the estimated relative pose of the instance of the object in the second image; The method of claim 1 further comprising:

16. handle, a) the landmarks projected using the estimated relative pose of the instance of the object in the second image; and b) the second plurality of detected landmarks; and comparing features of the determining whether a threshold level correspondence exists between the compared features of the corresponding projected landmark and the first plurality of detected landmarks; The method of claim 15 further comprising:

17. the image of the instance is a two-dimensional image; the detected landmarks are two-dimensional landmarks; the dimensional space of the received image is a two-dimensional space; the features of the corresponding projected landmark and the second plurality of detected landmarks being compared are position features in two-dimensional space; The method of claim 1.

18. detecting landmarks of an instance of an object in an image of the instance of the object; estimating a pose of the instance of the object relative to an imaging device that acquired the image; using the estimated pose to project landmarks from a three-dimensional model of the object into the three-dimensional space of the image of the instance of the object; identifying corresponding projected and detected landmarks; comparing features of the corresponding projected landmarks and the detected landmarks in the dimensional space to identify misalignments of the corresponding landmarks; identifying a subset of the corresponding landmarks with deviations greater than a threshold condition; determining a deformation, movement, or blurring of the instance of the object or a portion of the instance of the object based on the determined subset; A method comprising:

19. and identifying the deformation, the movement, or the blurring includes identifying a damaged portion of the instance of the object.

20. The method of claim 18.

20. Identifying the deformation, the movement, or the blurring, detecting the landmarks of the instance of the object in a second image of the instance of the object; determining the deformation, the movement, or the blur based on positions of the landmarks detected in the second image; and 20. The method of claim 18, comprising:

21. identifying a cluster of spatially close landmarks of the subset of corresponding landmarks with misalignment; determining the deformation or the movement of the object based on the clusters of spatially nearby landmarks; 20. The method of claim 18, further comprising:

22. and identifying the corresponding landmarks includes comparing contexts of the projected and detected landmarks to identify the corresponding landmarks.

20. The method of claim 18.

23. the image of the instance is a two-dimensional image; the detected landmarks are two-dimensional landmarks; the dimensional space of the received image is a two-dimensional space; the features compared to identify the offset are two-dimensional position features; 20. The method of claim 18.

Images