Processing images of objects and parts of objects, including multi-object constructs and deformed objects.

Abstract

A method of image processing may include receiving an image of an instance of an object and a three-dimensional model of the object, detecting a first plurality of landmarks of the instance of the object in the two-dimensional image, estimating a pose of the instance of the object in the received image relative to an imaging device that captured the image, where a relative pose in the received image is estimated from the first plurality of detected landmarks, projecting landmarks from the three-dimensional model of the object into a two-dimensional space of the received image of the instance of the object using the estimated relative pose, comparing features of corresponding projected landmarks and the first plurality of detected landmarks in the two-dimensional space, and determining whether a threshold level positional correspondence exists between the positions of corresponding projected landmarks and the first plurality of detected landmarks.

Description

Technical Field

[0005]

[0001] Cross - reference to Related Applications This application claims the benefit of priority of Greek Application No. 20210100909 filed on December 23, 2021 and U.S. Patent Application No. 17 / 654,647 filed on March 14, 2022. The entire content of the same is incorporated herein by reference.

[0002] This specification relates to the processing of objects and object parts, including multi - object constructs and deformed objects.

Background Art

[0003] Image processing is a certain type of signal processing where the signal to be processed is an image. An input image can be processed, for example, to generate an output image or a characterization of the image.

[0004] An example of image processing is pose estimation. As will be described in detail below, pose estimation is the process by which the relative position and orientation of an imaging device and an object are estimated from a two - dimensional image. Pose estimation can be based on other results of image processing. One example is landmark recognition. In landmark recognition, a two - dimensional image is processed to identify landmarks and their positions in the image. The identification information and positions of the landmarks are examples of results on which pose estimation can be based.

Summary of the Invention

[0005] In a first embodiment, the image processing method includes receiving an image of an instance of an object and a three-dimensional model of the object; detecting a first set of landmarks of the instance of the object in the two-dimensional image; estimating the pose of the instance of the object in the received image with respect to an imaging device that acquired the image, wherein the relative pose in the received image is estimated from the first set of detected landmarks; projecting landmarks from the three-dimensional model of the object onto the dimensional space of the received image of the instance of the object using the estimated relative pose; comparing the features of the corresponding projected landmarks with those of the first set of detected landmarks in dimensional space; and determining whether a threshold-level positional correspondence exists between the positions of the corresponding projected landmarks and the first set of detected landmarks.

[0006] Embodiments of the first, second, or third aspect may include one or more of the following features: The method may include detecting multiple landmarks of an instance of an object in an image, wherein the detected multiple landmarks include more landmarks than a first multiple landmarks, and selecting the first multiple landmarks from among the detected multiple landmarks. The selection of the first multiple landmarks may be random. The selection of the first multiple landmarks may be guided by the characteristics of the landmarks, which may be how certain it is that a given landmark has been properly detected.

[0007] This method, in response to determining that no threshold-level positional correspondence exists, re-estimates the relative pose of an object instance in a first image from a second set of landmarks of the object instance detected in the first image, wherein at least some of the landmarks among the second set of landmarks are different from the landmarks among the first set of landmarks; uses the re-estimated relative pose to project landmarks from a three-dimensional model of the object onto the dimensional space of the received image of the object instance; compares the features of the corresponding projected landmarks with those of the second set of detected landmarks in dimensional space; and determines whether or not a threshold-level positional correspondence exists between the positions of the corresponding projected landmarks and the second set of detected landmarks.

[0008] The method may further include selecting a second set of landmarks for an instance of an object. The second set of landmarks may be selected, for example, by identifying the landmarks in the first set of landmarks that have a relatively large positional difference from the corresponding projected landmark, and by excluding the landmarks in the first set of landmarks that have a relatively large positional difference from the second set of landmarks. The second set of landmarks may be selected, for example, by identifying the positions of the two-dimensional landmarks in the first set of landmarks that have a relatively large positional difference from the corresponding projected landmark and the detected landmark, and by excluding the landmarks in the first set of landmarks that are in the vicinity of the corresponding projected landmark and the detected landmark that have a relatively large positional difference from the second set of landmarks. The second set of landmarks may be selected, for example, by identifying the direction of misalignment between the corresponding projected landmark and the detected landmark, and by excluding the landmarks in the first set of landmarks that have a direction of misalignment different from most of the misalignment directions between the corresponding projected landmark and the detected landmark.

[0009] This method may include, in response to determining the existence of a threshold-level positional correspondence, identifying a subset of landmarks of detected object instances in an image with a relatively large displacement from the corresponding projected landmark, and drawing conclusions about the object instances based on the detected subset of landmarks. The conclusions may specify the deformed or damaged portion of the object instance. The conclusions may quantify the magnitude of the relatively large displacement, the direction of the relatively large displacement, or both the magnitude and direction.

[0010] By combining the positional differences between multiple corresponding projected landmarks and detected landmarks, and by comparing the combination of positional differences with a threshold condition, it can be determined that a threshold-level positional correspondence exists. The relative pose of an object instance can be estimated by forming a first estimation result of the relative pose of the object instance in the received image, identifying that the quality of the first relative pose estimation result is insufficient, and in response, forming a second estimation result of the relative pose of the object instance in the received image.

[0011] This method may include receiving a second image of an instance of an object, detecting a second set of landmarks of the instance of the object in the second image acquired by a second imaging device, estimating the pose of the instance of the object in the second image relative to the second imaging device that acquired the second image, wherein the relative pose in the second image is estimated from the second set of detected landmarks, and projecting landmarks from a three-dimensional model of the object into the dimensional space of the second image using the estimated relative pose of the instance of the object and the estimated relative pose of the instance of the object in the second image.

[0012] The method may further include comparing the features of a) a projected landmark using the estimated relative pose of an instance of an object in a corresponding second image with those of a second set of detected landmarks in a two-dimensional space, and determining whether a threshold-level correspondence exists between the compared features of the corresponding projected landmark and the first set of detected landmarks. The image of the instance may be a two-dimensional image. The detected landmarks may be two-dimensional landmarks. The dimensional space of the received image may be a two-dimensional space. The features of the corresponding projected landmark and the second set of detected landmarks being compared may be positional features in a two-dimensional space.

[0013] In a second embodiment, the method includes detecting landmarks of an instance of an object in an image of the instance of an object; estimating the pose of the instance of an object relative to the imaging device that acquired the image; projecting landmarks from a three-dimensional model of the object onto the dimensional space of the image of the instance of an object using the estimated pose; identifying corresponding projected landmarks and detected landmarks; comparing the features of corresponding projected landmarks and detected landmarks in dimensional space to identify deviations of corresponding landmarks; identifying subsets of corresponding landmarks with deviations greater than a threshold; and identifying deformation, motion, or obscuration of an instance of an object or a portion of an instance of an object based on the identified subsets.

[0014] Embodiments of the second, first, or third embodiment may include one or more of the following features: Identifying deformation, movement, or obscuration may include identifying damaged parts of an instance of an object. Identifying deformation, movement, or obscuration may include detecting landmarks of the instance of an object in a second image of the instance of the object and identifying deformation, movement, or obscuration based on the location of the landmarks detected in the second image. The method may include identifying spatially close clusters of landmarks of a subset of corresponding landmarks with displacement and identifying deformation or movement of an object based on clusters of spatially close landmarks.

[0015] Identifying a corresponding landmark may involve comparing the contexts of the projected and detected landmarks in order to identify the corresponding landmark. The image of the instance may be a two-dimensional image. The detected landmark may be a two-dimensional landmark. The dimensional space of the received image may be a two-dimensional space. The features compared to identify the shift may be two-dimensional positional features.

[0016] In a third aspect, a method for identifying differences between two-dimensional images may include receiving a first two-dimensional image and a second two-dimensional image, each of which includes at least a portion of the same instance of an object; receiving a three-dimensional model of the object; detecting a first plurality of two-dimensional landmarks of the instance of the object in the first image and a second plurality of two-dimensional landmarks of the instance of the object in the second image; estimating the pose of the instance of the object in each of the first and second images to one or more imaging devices that acquired the first and second two-dimensional images; projecting landmarks from the three-dimensional model of the object into two-dimensional space using at least one of the estimated relative poses; comparing the positions in two-dimensional space of the corresponding projected landmarks and the detected two-dimensional landmarks in each of the first plurality of detected two-dimensional landmarks and the second plurality of detected two-dimensional landmarks; and identifying an anomaly landmark among the first plurality of two-dimensional landmarks or among the second plurality of two-dimensional landmarks based on the comparison.

[0017] Other embodiments of the methods described above in the first, second, and third embodiments may include corresponding systems and apparatus configured to perform the actions of the method, and computer programs tangibly embodied in a machine-readable data storage device and constituting a data processing device for performing the actions.

[0018] Details of one or more embodiments are described in the accompanying drawings and the following description. Other features and advantages of the embodiments will become apparent from the description and drawings and from the claims. [Brief explanation of the drawing]

[0019] [Figure 1] Figure 1 is a schematic diagram of acquiring a collection of different images of an object. [Figure 2]FIG. 2 is a schematic diagram of a set of two-dimensional images acquired by one or more cameras. [Figure 3] FIG. 3 is a flowchart of a process performed by a computer to process a two-dimensional image of an object. [Figure 4] FIG. 4 is a schematic diagram of an embodiment of a part of the process of FIG. 3 performed using two different images of two different instances of the same object. [Figure 5] FIG. 5 schematically represents the selection of different subsets of two-dimensional landmarks during the performance of a part of the process of FIG. 3. [Figure 6] FIG. 6 is a schematic diagram of an embodiment of a part of the process of FIG. 3 performed using three different images of four different objects. [Figure 7] FIG. 7 is a histogram representing the misalignment between landmarks detected from a two-dimensional image and landmarks projected from a three-dimensional model. [Figure 8] FIG. 8 is a flowchart of a process performed by a computer to annotate landmarks appearing in a 3D model. DETAILED DESCRIPTION OF THE INVENTION

[0020] Like reference numerals in the various drawings indicate like elements.

[0021] FIG. 1 is a schematic diagram of the acquisition of a set of different images of an object 100. For purposes of illustration, object 100 is shown as an assembly of geometric parts (e.g., cubes, polyhedra, parallelepipeds, etc.) that are not ideally marked. However, in actual applications, objects generally have more complex shapes and are, for example, textured or otherwise marked with decorative decorations, wear marks, or other markings on the underlying shape.

[0022] A collection of one or more imaging devices (shown in this example as cameras 105, 110, 115, 120, 125) can be arranged continuously or simultaneously at different relative positions around an object 100 and can be oriented at different relative angles with respect to the object 100. The positions can be distributed in three-dimensional space around the object 100. The orientation can further vary in three dimensions, i.e., all of the Euler angles (or yaw, pitch, and roll) can vary. The relative arrangement and orientation of cameras 105, 110, 115, 120, 125 with respect to the object 100 can be referred to as the relative pose between the camera and the object. Since cameras 105, 110, 115, 120, 125 take different relative poses, each of cameras 105, 110, 115, 120, 125 acquires a different image of the object 100.

[0023] The relative pose between the camera and the object can be defined in different reference coordinate systems. For example, the reference coordinate system for the relative pose of the camera and the object can be defined based only on the camera and the object, for example, by drawing a straight line between a point on the object and a point on the camera and by selecting a point along this line. The length of this line defines the distance between the object and the camera, and the line can be used to define the angular inclination between the camera and the object. As another example, the reference coordinate system can be defined with respect to other reference points, such as a position on the ground or another location, as an example. The distances and orientations defined with respect to these points can be converted to distances and orientations in a reference coordinate system defined based only on the camera and the object.

[0024] Returning to Figure 2, even a simplified object such as object 100 contains many landmarks 130, 131, 132, 133, 134, 135, 136, ... A landmark is a location of interest on object 100. Landmarks can be located at a geometric position on the object or on an underlying geometric marking. As will be described in detail below, landmarks can be used to identify the pose of an object. Landmarks can also be used for other types of image processing, for example, to classify objects, to extract features from objects, to locate other structures (geometric structures or markings) on an object, to assess damage to an object, and / or to serve as an origin from which measurements can be taken by these and other image processing techniques.

[0025] Figure 2 is a schematic diagram of a collection of two-dimensional images 200 acquired by one or more cameras, such as cameras 105, 110, 115, 120, and 125 (Figure 1). The images in the collection 200 show the object 100 in different relative poses. Landmarks such as landmarks 130, 131, 132, 133, 134, 135, 136, ... appear in different positions in different images, if they were to appear. For example, in the leftmost image of the collection 200, landmarks 133 and 134 are obscured by the rest of the object 100. In contrast, in the rightmost image 210, landmarks 131, 135, and 137 are obscured by the rest of the object 100.

[0026] Figure 3 is a flowchart of a process 300 performed by a computer for processing a two-dimensional image of an object. The two-dimensional image may include arrangements of multiple objects and / or deformed objects. Process 300 may be part of a process for identifying, for example, parts of individual deformed objects, objects or parts of objects that have moved from one image to the next, and / or parts of objects that are obscured in the image. Process 300 may be performed by one or more data processing devices that perform data processing activities. The activities of process 300 may be performed according to a machine-readable instruction set, a hardware assembly, or logic of these instructions and / or other combinations of instructions.

[0027] In step 305, the device performing step 300 receives an image of the object and a three-dimensional model of the object in the image.

[0028] The received image can be acquired by any of many different types of cameras or other imaging devices. For example, the image can be acquired by a smartphone, digital camera, medical imaging device, LIDAR camera, X-ray machine, etc. In some embodiments, a single received image combines different types of information, such as information acquired by multiple imaging devices or information acquired using different imaging mechanisms. For example, a single received image may combine information acquired from different poses (stereoscopic imaging), information acquired from the same pose but using different dynamic ranges (high dynamic range imaging), information acquired using a polarization filter, etc. The received image may therefore contain two-dimensional information, three-dimensional information, and higher-dimensional information. The information may include stereoscopic information, polarization information, high dynamic range information, depth scanning information (e.g., LIDAR), polarization and other filters, masks, labels related to pixel content, vector field information of transitions (e.g., color, shape), motion information, etc. In some examples, the information is acquired using a single imaging device (e.g., a stereoscopic camera). In other examples, post-acquisition processing is performed to combine information acquired using multiple imaging devices.

[0029] In some embodiments, the device performing step 300 itself acquires the received image. In other embodiments, the image is received from an imaging device directly or via one or more intermediate devices. For example, the image may be communicated to the device performing step 300 using wired or wireless data communication, either as a standalone image or as part of a video stream.

[0030] A three-dimensional (3D) model generally represents an object in three-dimensional space, detached from any reference coordinate system. 3D models can be generated manually, algorithmically (through procedural modeling), or by scanning a real object. Surfaces in a 3D model can be defined using texture mapping. In some examples, a 3D model of an object can be generated as an assembly of its constituent parts or components (e.g., using computer-aided design (CAD) software). For example, a 3D model of a car may be formed as an assembly of 3D CAD models of the car's constituent parts, or a 3D model of a mouth may be formed as an assembly of models of the palate and tooth crowns within the mouth. However, in other cases, a 3D model may begin as a whole that is subdivided into constituent parts. For example, a 3D model of an organ may be subdivided into various constituent parts based on instructions from a medical or other professional.

[0031] This disclosure refers to three-dimensional models of “objects” or “the same object,” but a three-dimensional model is generally not a model of one physical instance of an object. Rather, a three-dimensional model is generally a general, ideal model of different objects that share common characteristics. One example is a three-dimensional model of an automobile or electrical equipment that does not consider the details of a particular instance of that automobile or electrical equipment—a three-dimensional model of an automobile or electrical equipment of a particular manufacture and model. Another example is a three-dimensional model of different organs or teeth of an individual with specific physiological and / or demographic characteristics (e.g., age, sex, height, weight, jaw width, etc.).

[0032] In some examples, the image and 3D model are of the same part of an object. In some examples, the image may contain multiple objects (or parts of multiple objects), and multiple three-dimensional models may be received. The exact nature of the object—or part of an object—may depend on the context of its use. Illustrative objects include cars, internal organs, teeth, and objects in a landscape (e.g., houses, streets, lampposts, rivers, etc.). Considering these illustrative objects, the illustrative parts of an object are: - Automotive parts (e.g., bumpers, wheels, body panels, hoods, windshields, and side panels), - Parts of organs (e.g., chambers, valves, cavities, lobes, tubes, membranes, and vascular structures), - Tooth parts (e.g., crown, gingiva, root), - This includes parts of objects within a landscape (e.g., house roofs, intersections, river bends, etc.). Other objects and other parts of objects would be more appropriate in the context of other uses. For this diversity and for the sake of brevity, objects or parts of objects are collectively referred to as "objects" in this specification.

[0033] The device performing process 300, or one or more other devices, may use any of many different approaches to ensure that the received image and 3D model are of the same object. For example, in some embodiments, metadata associated with the received image may characterize the object in the image. For example, the year of manufacture and model may be associated with a 2D image of a car. A patient name, or physiological and demographic characteristics, may be associated with a medical or dental image. GPS coordinates may be associated with a landscape image. Such metadata may be used to identify a 3D model of the same object. For example, an existing library of 3D car models may be searched for a 3D model of a car of the same manufacture and model. Demographic and / or physiological information may be used to find a 3D model representing that a patient has those demographic and / or physiological characteristics. In some examples, the 3D model may be obtained from the same instance of the object, and metadata associated with the image may be used. For example, patient names may be used to retrieve previously generated 3D images of a patient's physiological functions, or to ensure correspondence between medical images acquired using one imaging modality and 3D images acquired using a different three-dimensional imaging modality.

[0034] In step 310, the device performing step 300 detects landmarks in the received image of the object. Landmarks may be detected using, for example, a machine learning model. An exemplary machine learning model for landmark detection is detectron2, available at https: / / github.com / facebookresearch / detectron2. In some embodiments, the landmark detection machine learning model may generate a detection score or other characterization for each landmark indicating how certain the landmark was properly detected.

[0035] In step 315, the device performing step 300 selects a true subset of landmarks detected in the received image of the object. In some embodiments, the selection of landmarks is random. In other embodiments, the selection of landmarks is not random, and one or more parameters guide the selection of landmarks. For example, in some embodiments, the device performing step 300 may selectively select landmarks that have specific location features in the received image. For example, the device performing step 300 may selectively select landmarks that are relatively far from each other or that are relatively uniformly distributed across the field of view in the received image. As another example, in embodiments where a landmark detection machine learning model generates a characterization of how certain different landmarks are that they have been properly detected, the device performing step 300 may selectively select landmarks that have been detected with relatively high certainty. As yet another example, the selection of landmarks may be guided by a combination of location, certainty, and / or other parameters.

[0036] In step 320, the device performing step 300 estimates the relative pose of an object in the received image using selected landmarks. The relative pose may be estimated using, for example, a machine learning model. For example, a landmark detection-dependent pose estimation unit is the OpenCV function SolvePNP, described at https: / / docs.opencv.org / master / d7 / d53 / tutorial_py_pose.html.

[0037] As another example, relative pose can be estimated using forward prediction from a derived inverse model. An example is described in the publication "iNeRF: Inverting Neural Radiance Fields for Pose Estimation" by Yen-Chen Lin et al. (arXiv:2012.05877v3, August 10, 2021), the contents of which are incorporated herein by reference (available at https: / / api.semanticscholar.org / CorpusID:228083990).

[0038] In some embodiments, the quality of pose estimation may be scored or otherwise characterized. For example, in some embodiments, a binary valid / invalid characterization of pose estimation quality may be generated. A "valid" pose is of sufficient quality, while an "invalid" pose is of insufficient quality. Criteria for invalidating pose prediction may be established based on criteria that reflect the real-world conditions under which the received image may be acquired. The criteria may be adjusted according to the nature of the object. For example, for pose estimation where the object is a car, the following can be said: - The camera must be positioned at an altitude of 0 to 5 meters relative to the ground below the vehicle. - The camera must be within 20 meters of the vehicle. - The camera roll relative to the ground below the car is small (e.g., less than + / - 10 degrees). And, - The vehicle's boundaries must largely coincide with the vehicle's boundaries resulting from the predicted pose.

[0039] If the pose estimation does not meet these criteria, the pose prediction may be indicated as invalid.

[0040] As another example, in some embodiments, a non-binary, more granular characterization of the quality of pose estimation may be generated. For example, a further machine learning model may be used to detect the contours of an object in the received two-dimensional image. Furthermore, the estimated pose may be used to project a 3D model of the object to form a surrogate two-dimensional image of the object in the estimated pose. The contours detected from the received two-dimensional image may be compared to the contours of the object in the surrogate two-dimensional image to characterize the quality of pose estimation. In some embodiments, the results may characterize the quality of pose estimation based on the object as a whole. For example, correspondences between contours of the entire object in two images may be characterized. In other embodiments, the results may characterize the quality of pose estimation based on parts or regions of the object. For example, correspondences between contours of only parts or regions of a larger object may be characterized. Such contour comparisons may be further used to generate a binary valid / invalid characterization of the quality of pose estimation.

[0041] As yet another example, in some embodiments, the quality of pose estimation may be estimated from the context surrounding individual landmarks. For example, the context of a landmark detected in the received image may be compared to the context of what is considered the corresponding landmark in a surrogate image formed from a 3D model. For example, a 3D landmark of an object from a 3D model may be projected onto a two-dimensional or three-dimensional surrogate image in the estimated pose. Such landmark context may include, for example, the shape and / or visual characteristics of features around the landmark, such as color, structure, pattern, optical properties (e.g., reflectivity, polarization), and the size of typical adjacent structures. In some embodiments, these characteristics are learned from exemplary images or computed from a 3D model (e.g., a door handle is typically attached to a door, and the door is given a specific separation from other parts of the car, etc.).

[0042] As yet another example, in some embodiments, the quality of pose estimation may be characterized by estimating the pose multiple times and comparing different pose estimations. For example, different true subsets of landmarks detected in the received image of an object may be selected, for example, in 315. The relative pose of the object in the received image may be estimated multiple times using different true subsets to determine the stability of the different pose estimations, i.e., how much the different pose estimations deviate from each other. In some embodiments, for example, stability is evaluated using landmarks from a portion of the received image rather than the entire received image. For example, a portion may be defined to exclude landmarks in the object of interest or other highly variable portions of the received image. In such cases, the landmarks used to estimate the quality of the pose estimation are separated from other more variable landmarks. If the quality of the pose estimation is insufficient, the pose may be re-estimated until the quality is sufficient. For example, step 300 may return to 315 to select different subsets of landmarks and the pose re-estimated using the different subsets. In some cases, step 300 may be repeated to step 310 to detect further landmarks or to detect landmarks using a different approach, such as a different or finely tuned machine learning model. If the quality of pose estimation remains insufficient, step 300 may be stopped for a given received image, and a different image may be received and used.

[0043] In step 325, the device performing step 300 uses the three-dimensional model of the object received in step 305 and the relative pose estimated in step 320 to project landmarks from the three-dimensional model into the same dimensional space as the received image. Essentially, the calculations performed by the device performing step 300 orient and position the three-dimensional model to match the relative pose estimated in step 320. Three-dimensional landmarks in the visible three-dimensional model can be identified in the virtual received image formed in the same dimensional space as the received image.

[0044] After projection, the corresponding projected and detected landmarks can be identified. For example, the context of the projected and detected landmarks can be compared to identify the corresponding landmarks. Such landmark contexts may include, for example, the shape and / or visual characteristics of features around the landmark, such as color, structure, pattern, optical properties (e.g., reflectivity, polarization), and the size of typical adjacent structures. In some embodiments, these characteristics are learned from exemplary images or computed from 3D models (e.g., a door handle is typically attached to a door, and the door is given a specific separation from other parts of the car, etc.).

[0045] In step 330, the device performing step 300 compares the location or other characteristics of the landmarks projected onto the virtual image in step 325 with the location or other characteristics of the corresponding landmarks detected in the image received in step 310. The comparison is performed for each landmark in at least some of the dimensions onto which the three-dimensional model was projected in step 325.

[0046] The results of the comparison can be expressed in various different ways depending on the specific characteristics being compared. For example, suppose the 2D or 3D position of each individual landmark projected at 325 is compared with the 2D or 3D position of the corresponding individual landmark detected at 310. The 2D or 3D position difference can be expressed in terms of both magnitude and direction. The magnitude of the separation between landmarks can be expressed, for example, in units of pixels as a percentage of the width of the received image, or otherwise. As further examples, differences in color can be expressed in terms of wavenumber, differences in reflectivity in terms of radiometric units, and differences in polarization in terms of angular difference.

[0047] In some embodiments, the device performing step 300 may further generate values ​​characterizing combinations of differences of several corresponding landmarks. For example, the average difference of 2D or 3D positions for a set of corresponding landmarks, or the algebraic or vector sum of the differences of 2D or 3D positions, may be generated and used in subsequent activities.

[0048] In step 335, the device performing step 300 determines whether a threshold-level correspondence exists between the features of the landmarks projected onto the virtual image in step 325 and the features of the landmarks detected in the received image in step 310. Both the threshold level and the feature difference compared to the threshold level may be per landmark or per combination. For example, the number or proportion of corresponding landmarks with individual differences below a threshold condition may be used to determine whether a threshold-level correspondence exists. As another example, the vector sum of the 2D or 3D positional differences of several corresponding landmarks may be compared to the threshold-level correspondence to determine whether a threshold-level correspondence exists.

[0049] In either case, the threshold level may be expressed by objective terms independent of a particular example of process 300, or by subjective terms adjusted for a particular example of process 300. For example, in some embodiments, an objective threshold level—for example, a specific number of pixels or a percentage of the width of the received image—may be applied to multiple examples of process 300. In other embodiments, a subjective threshold level—for example, the standard deviation of the 2D or 3D position difference, or a value adjusted for the certainty that a landmark was properly detected in the received image—may be applied to different examples of process 300.

[0050] In response to determining that no threshold-level correspondence exists, the device performing step 300 selects a different true subset of landmarks detected in the received image (i.e., a different true subset of landmarks detected in 310) in 340. In some embodiments, landmarks are selected randomly. In other embodiments, the selection of landmarks is not random, but guided. In addition to the parameters described above in 315, the results of the landmark-by-landmark comparison in 330 may further guide the selection of a different subset of landmarks. For example, in some embodiments, if the difference between the landmark projected in 325 and the corresponding individual landmark detected in 310 is relatively small, that individual landmark may be selectively included in a different true subset. As another example, in some embodiments, if the difference between the landmark projected in 325 and the corresponding individual landmark detected in 310 is relatively large, that individual landmark may be excluded from a different true subset. For example, referring to histogram 700 (Figure 7), landmarks from bar cluster 720 may be selected for different proper subsets, while landmarks from bar 715 may be excluded from different proper subsets.

[0051] After selecting different proper subsets in 340, the device performing step 300 estimates a new relative pose of the object using the different proper subsets in 320, projects landmarks from the 3D model onto the dimensional space of the received image using the new relative poses in 325, and compares the positions of the landmarks in 330. This may be repeated until a threshold-level correspondence is determined to exist in 335.

[0052] In response to determining that a threshold-level correspondence exists, the device performing step 300 identifies an anomalous landmark among the landmarks detected in the received image at 345. The anomalous landmark may be detected at 310 from all of the markers detected, at 335 from the landmarks in the subset that provides the threshold-level correspondence, or at 310 from a different subset of the landmarks detected.

[0053] Anomaly landmarks can be detected by many different methods. For example, in some embodiments, a histogram of the differences between corresponding landmarks (e.g., histogram 700, Figure 7) can be generated and used to identify anomalies. As another example, threshold differences can be used to identify anomalies. In some embodiments, threshold differences can be expressed by objective terms (e.g., exceeding a certain number or percentage of pixels in the width of the received image). In some embodiments, threshold position differences can be expressed by subjective terms, for example, terms relative to a specific example of process 300. For example, the standard deviation from the mean difference, or a direction related to the 2D or 3D directional difference of other corresponding landmarks in a specific example of process 300, can be used to identify anomalies.

[0054] Identified anomalous landmarks can be applied to various different activities depending on the operational context. For example, in an operational context where damage or deformation of an object instance is identified, anomalous landmarks can be used to identify the damaged or deformed portion of the object instance. For instance, a cluster of spatially close anomalous objects may indicate that the underlying portion of the object instance is damaged or deformed in its vicinity. As another example, anomalous landmarks can be used to characterize the degree of damage or deformation. For example, the magnitude of a positional difference—as well as color or other optical differences—can be obtained as an indicator of the degree of damage or deformation. If the difference is relatively small, this may be interpreted as an indication of normal wear and tear and damage to the object instance. On the other hand, if the difference is relatively large, this may be interpreted as an indication of more severe damage to the object instance.

[0055] As another example, in a motion context where the movement of a part of an object instance is identified, anomalous landmarks may be used to characterize the movement. For example, the magnitude and direction of a positional difference may be obtained as indicators of the magnitude and direction of the movement. Examples of such motion contexts include, for example, contexts where the movement of a robot's movable arm or other joint or part of another automated machine is identified.

[0056] As yet another distinct example, in a behavioral context where obscurity of a portion of an object instance is identified, anomalous landmarks may be used to characterize the obscured portion. For example, if the obscurity is decorative or even a new paint coating, color or other optical differences may be used. When 2D or 3D positional differences are used to identify obscurity, anomalous landmarks may be anomalous in the sense that they are not detected in the received image. In some embodiments, the location of anomalous landmarks may be used to define the rough boundary of the obscured body. In some embodiments, multiple executions of step 300 using different images may be used to characterize the movement of the obscured body between images, since different landmarks will be anomalous because they are not detected.

[0057] As another example, in some behavioral contexts, deformation of a soft body is identified. In such contexts, anomalous landmarks may be used, for example, to characterize the deformation of an object using a "wireframe" 3D model in order to establish the kinematic arrangement of a wireframe using anomalous landmarks.

[0058] As another example, in certain behavioral contexts, object growth is identified. In such contexts, the nature of anomalous landmarks can be used to identify the type of growth. For example, if the volume of an object is increasing in three dimensions, the 2D positional difference of the corresponding two-dimensional landmarks may reflect their distance from their reference positions in the object. As another example, if an object is increasing (e.g., lengthening) in only one dimension, the 2D positional difference may reflect the distance of the object from its reference line or plane. In such cases, "anomalous" landmarks may comprise a relatively large proportion—or even the majority—of the landmarks.

[0059] Figure 4 is a schematic diagram of an exemplary embodiment of a portion of step 300, which is performed using two different two-dimensional images of two different instances of the same object. As described above, the images received in 305 may contain information in multidimensional space, and comparisons may be made in such dimensions. However, for illustrative purposes, an exemplary embodiment of a portion of step 300 is performed using a first two-dimensional image 405 and a second two-dimensional image 410. Furthermore, all comparisons are in two-dimensional space.

[0060] Image 405 is a 2D image of an instance of object 415 that has been deformed to a relatively small degree. Image 410 is a 2D image of an instance 420 of the same object that has been deformed to a relatively larger degree. Both object instances 415 and 420 are represented by the same 3D model 425. In particular, the 3D model 425 is a general and ideal representation of object instances 415 and 420 in three-dimensional space. For example, the 3D model 425 could be a CAD model or process model of an object without any deformation or obscuration.

[0061] As described above, the 2D images 405, 410 and the 3D model 425 are received in 305 (Figure 3) by one or more data processing devices that perform data processing activities. In the schematic diagram, a collection of three-dimensional landmarks 430 is shown in the 3D model 425 as an arrangement of unfilled points at different locations in the 3D model 425. The three-dimensional landmarks 430 are points of interest in the 3D model 425. In some embodiments, the three-dimensional landmarks 430 are identified in metadata that accompanies the 3D model 425 when it is received in 305 (Figure 3). In other embodiments, processing performed by a computer may be used to annotate the 3D model 425 with landmarks. An example of such processing is step 800 (Figure 8), which is described below.

[0062] The receiving device detects a collection of two-dimensional landmarks 435 in images 405 and 410 in 310 (Figure 3). For illustrative purposes, the 2D landmarks 435 are shown as a two-dimensional arrangement of filled black dots at different locations along the dashed contours of object instances 415 and 420. In a real-world embodiment, such arrangement is unnecessary, and the location of the 2D landmarks 435 may be expressed using 2D position coordinates or otherwise. In the schematic diagram shown, the 2D landmarks 435 are features of corners or other edges in object instances 415 and 420. This is not necessarily the case. The 2D landmarks 435 may be located in other places in object instances 415 and 420. For example, the 2D landmarks 435 may be located at the joints between different components, on decorative features on the surface of object instances 415 and 420, or in other places in object instances 415 and 420.

[0063] The receiving device further selects subsets of 2D landmarks 435 within each set of 2D landmarks 435 in 315 (Figure 3). For illustrative purposes, the subsets of 2D landmarks 435 are selected from similar contiguous areas 440 in both images 405, 410. However, this is not necessarily the case. The selected 2D landmarks 435 may be distributed across images 405, 410—randomly or otherwise—so that there is no such thing as an “area” in which the landmarks are selected.

[0064] Using a selected subset of 2D landmarks 435, the device estimates the relative poses of object instances 415 and 420 in images 405 and 410 in 320 (Figure 3). With respect to the selected subset of 2D landmarks 435 from image 405, they are observed in portions of object instance 415 that are undeformed or only slightly deformed. For example, 2D landmark 450 is observed near portions of object instance 415 that are only slightly deformed. In contrast, with respect to the selected subset of 2D landmarks 435 from image 410, 2D landmarks 435 are observed in portions of object instance 420 that are only slightly deformed. For example, 2D landmarks 455, 460, and 465 are observed near portions of object instance 415 that are only slightly deformed.

[0065] The estimated pose of object instance 415 is relatively accurate because the 2D landmarks 435 in the selected subset from image 405 are located where they are expected to be. Any error in the estimation due to the 2D landmarks 450 is relatively small. In contrast, the estimated pose of object instance 420 is relatively inaccurate because the 2D landmarks 435 in the selected subset from image 410 are located significantly far from where they are expected to be. Indeed, in some cases, pose estimation may return unacceptably inaccurate results, or even fail to return results at all, and the pose may be re-estimated, for example, using a different subset of detected landmarks or using detected landmarks with a different or finely tuned machine learning model.

[0066] Using the estimated relative pose, the device projects 3D landmarks from the 3D model 425 onto a virtual two-dimensional image in 325 (Figure 3). For the pose estimated from image 405, this yields a collection of landmarks 470. The landmarks in collection 470 are "two-dimensional" in that they are located only in two-dimensional space (in contrast to the landmarks 430 in three-dimensional space in the 3D model 425). Although the landmarks in collection 470 are two-dimensional, they are further shown in collection 470 as unfilled points to indicate their correspondence to the 3D landmarks 430 in the 3D model 425.

[0067] In Figure 330, the device compares the position of the 2D landmark 435 detected in image 405 with the position of the 2D landmark in aggregate 470. This comparison is schematically represented in the lower left corner of Figure 4. Where the 2D landmark in aggregate 470 overlaps with or nearly overlaps with the 2D landmark 435, an unfilled point is superimposed with "x". However, where the 2D landmark 435 and the 2D landmark in aggregate 470 are displaced to a degree that allows for drawing, the 2D landmark 435 is represented by a filled black point. In the embodiment shown, two different 2D landmarks 435—namely, 2D landmark 450 and 2D landmark 475—are displaced to a degree that allows for drawing. The corresponding 2D landmarks in aggregate 470 are represented by an unfilled point without using "x".

[0068] Based on a comparison of the position of 2D landmark 435 detected in image 405 with the position of 2D landmarks in aggregate 470, the device determines in 335 (Figure 3) whether a threshold level correspondence has been reached. Furthermore, the device identifies anomalous landmarks (e.g., 2D landmarks 450, 475) in 345 (Figure 3).

[0069] In the embodiments shown, the cluster of landmarks formed by projecting the pose estimated from image 410 is not shown. This is merely a schematic diagram. A cluster of landmarks may be formed, and the position of the 2D landmark 435 detected in image 410 can be compared with the position of the 2D landmark in such a cluster. However, the result of such a comparison will be a larger discrepancy between the positions of the landmarks. In fact, in some embodiments, it may be difficult to identify the corresponding landmarks.

[0070] When faced with a positional misalignment or other indication of inaccuracy greater than the threshold level correspondence, the device selects a different subset of landmarks 435 from image 410 in 335 (Figure 3). In 320 (Figure 3), the relative pose of object 420 may again be estimated using the selected subset; in 325 (Figure 3), the 3D landmark may be projected onto the 2D image from the 3D model 425 using the estimated relative pose; and in 330 (Figure 3), the position of the 2D landmark is compared to the position of the 2D landmark formed by projecting the 3D landmark. This process may be repeated until the positional misalignment is no longer greater than the threshold level correspondence.

[0071] Figure 5 schematically illustrates the selection of different subsets of 2D landmarks 435 detected in image 410, which may prove suitable for accurately estimating the relative pose of object instance 420. In particular, the selected landmarks are located in area 505, which encompasses portions of object instance 420 that are undeformed or only slightly deformed. Since the 2D landmarks 435 in area 505 are located where they are expected to be, the estimated pose of object instance 420 may be relatively accurate.

[0072] As mentioned above, in 330 (Figure 3), the position of the 2D landmark 435 detected in image 410 can be compared to the position of the 2D landmark projected from the 3D model 425. This comparison is schematically represented at the bottom of Figure 5, where the overlapping projected and detected objects are indicated using unfilled dots superimposed with "x". The remainder of the detected 2D landmark 435 is represented by filled black dots. The remainder of the projected 3D landmark is represented by unfilled dots.

[0073] Figure 6 is a schematic diagram of some embodiments of process 300 performed using three different images 605, 610, and 615 of four different objects 620, 625, 630, and 635. Again, images 605, 610, and 615 are shown in two dimensions for illustrative purposes. Images 605, 610, and 615 differ from each other in that the pose of at least one object 620, 625, 630, and 635 differs in the images. In the embodiments shown, the pose of object 625 differs in images 605, 610, and 615. In other embodiments, the poses of one or more other objects 620, 630, and 635 may differ—whether or not the pose of object 625 also differs.

[0074] In some embodiments, the device performing step 300 may receive separate 3D models of each of the objects 620, 625, 630, and 635. In other embodiments, the device performing step 300 may receive only one 3D model of any of the objects 620, 625, 630, and 635. For example, the device may receive only the 3D model 425 of object 620 (Figure 4).

[0075] The receiving device detects the collection of two-dimensional landmarks 640, 645, and 650 in images 605, 610, 615, and 410 in 310 (Figure 3). For illustrative purposes, the 2D landmarks 640, 645, and 650 are also shown as a two-dimensional arrangement of filled black dots at different locations along the dashed outlines of object instances 620, 625, 630, and 635.

[0076] The receiving device, in 315 (Figure 3), further selects subsets of 2D landmarks 660, 665, and 670 from the sets of 2D landmarks 640, 645, and 650, respectively. For illustrative purposes, the subsets of 2D landmarks 660, 665, and 670 are selected from similar areas 655 within images 605, 610, and 615. However, the selected 2D landmarks may be further distributed across images 605, 610, and 615.

[0077] Using selected subsets 660, 665, 670 of 2D landmarks 640, 645, 650, and one or more corresponding 3D models, the device estimates the relative pose of at least one object instance 620, 625, 630, 635 in images 605, 610, 615 in 320 (Figure 3).

[0078] In some examples, the subsets 660, 665, and 670 of landmarks originate from at least one object 620, 625, 630, or 635, and the estimated poses of objects 620, 625, 630, and 635 are relatively accurate. For example, referring to subsets 660 and 670, a relatively large number of landmarks are selected from object 635. Assuming a 3D model of object 635 was available, the relative pose of object 635 can be estimated with relative accuracy.

[0079] In other cases, the estimated poses of objects 620, 625, 630, and 635 are relatively inaccurate. For example, - Too few landmarks were selected from objects 620, 625, 630, and 635 for which corresponding 3D models exist. - The parts of objects 620, 625, 630, and 635 for which corresponding 3D modes exist are unclear, or, - Difficulty in assigning landmarks to corresponding 3D models (e.g., due to damage or deformation), There are several possible contributions to this inaccuracy, including the following.

[0080] For example, referring to subset 645, even if a relatively large number of landmarks are selected from object 620, the pose of object 625 obscuring parts of object 620 can cause pose estimation based on object 620 to be relatively inaccurate. In fact, in some cases, pose estimation may return unacceptably inaccurate results, or even no results at all.

[0081] Using each of the estimated relative poses, the device projects 3D landmarks from one or more 3D models onto a 2D image in 325 (Figure 3) and obtains each set of landmarks. In 330 (Figure 3), the device further compares the positions of 2D landmarks 640, 645, and 650 detected in images 605, 610, and 615 with the positions of the 2D landmarks projected from the 3D models.

[0082] At this point, assuming that a threshold level correspondence has been reached, the projection of 3D landmarks from the 3D model onto the 2D image in 325 can be used to identify anomalous landmarks using various different methods. For example, referring to image 610, a comparison of the location of 2D landmark 645 with the location of the 2D landmark projected from the 3D model of object 620 (e.g., 3D model 525, Figure 4) shows that many landmarks from object 620 were not detected in image 610. If no landmarks from object 620 are detected at all, this can be interpreted as an indication that object 620 is partially obscured in image 610. Furthermore, a rough outline of the obscured area can be formed based on the locations of the undetected landmarks.

[0083] As another example, the positions of 2D landmarks 640, 645, and 650 identified in images 605, 610, and 615 can be compared to each other to determine, for example, that the relative pose of an object has changed. For example, referring to images 605 and 615, a comparison of the positions of 2D landmarks 640 and 650 shows that the positions of the 2D landmarks from objects 620, 630, and 635 have not changed substantially. In contrast, the position of the 2D landmark from object 625 has not changed substantially, but it shows that the pose of object 625 in images 605 and 615 is different.

[0084] Figure 7 shows a histogram 700 representing the discrepancy between landmarks detected from the received image and landmarks projected from the 3D model. Devices implementing the methods described herein generally do not form and display histograms like histogram 700 itself, but histogram 700 illustrates how the discrepancies between corresponding landmarks may be at play in the various activities of these methods. As mentioned above, the discrepancy may be a 2D positional discrepancy, a 3D positional discrepancy, or a discrepancy in yet another different dimension (such as color, reflectance, or polarization).

[0085] Histogram 700 includes a horizontal axis 705 and a vertical axis 710. The horizontal axis 705 is divided into many intervals, each encompassing a range of deviations between corresponding detected and projected landmarks. For example, one such interval encompasses zero deviation between corresponding landmarks—as is the case when the corresponding landmarks are identical. The positions along the vertical axis 710 represent the number of corresponding landmarks with deviations within each interval. Bars extending far upward from the horizontal axis 705 indicate that the number of corresponding landmarks within the deviation range of the horizontal axis 705 spanning that bar is greater than the number of landmarks in bars that do not extend so far from the horizontal axis 705. For example, for most of the deviation range of the horizontal axis 705, the number of corresponding landmarks appears to be zero. However, a recognizable number of corresponding landmarks have deviations within the range encompassed by bar 715. Furthermore, a relatively large number of corresponding landmarks have deviations within the range encompassed by bars in cluster 720.

[0086] As described above, histogram 700 can show how positional and other shifts between corresponding landmarks can affect various activities in these methods. For example, suppose histogram 700 represents the positional shifts of a first subset of corresponding landmarks resulting from the comparison in, for example, 330 (Figure 3). Corresponding landmarks with positional shifts within the range encompassed by bar 715 can lower the average correspondence below the threshold level. For example, this can be due not only to relatively large positional shifts of the corresponding landmarks themselves, but also to landmarks detected from the received image, making the estimated relative pose in 325 (Figure 3) more inaccurate.

[0087] In this situation, the guided selection of different subsets of detected landmarks in 340 (Figure 3) may exclude landmarks with displacements from the projected landmarks within the range enclosed by bar 715. Furthermore, detected landmarks with displacements within the range enclosed by the bars in cluster 720 may be selectively selected. In some situations, other detected landmarks (i.e., landmarks with displacements that do not appear in histogram 700) may also be selected.

[0088] Such guided selections can be repeated multiple times, with each iteration improving the accuracy of relative pose estimation. For example, bars within cluster 720 may appear to be very close together on the horizontal axis 705 of the scale of existence, but a change in scale may indicate that other landmarks with displacements within the range encompassed by the bars in cluster 720 must be excluded from the next subset.

[0089] As another example of how misalignments between corresponding landmarks may function in the manner described herein, suppose that histogram 700 represents all misalignments among the corresponding landmarks, as can be identified, for example, after a threshold level correspondence is reached in 335 (Figure 3). In this case, corresponding landmarks with misalignments within the range encompassed by bar 715 may be identified as anomalous and can serve as grounds for drawing conclusions about the object instance in the image in which the landmark was detected. For example, a detected landmark with a 2D or 3D positional misalignment within the range encompassed by bar 715 may be identified from a deformed or damaged portion of an object instance. As another example, a detected landmark with a 2D or 3D positional misalignment within the range encompassed by bar 715 may be identified from a portion of an object instance that has moved between images. As yet another example, a detected landmark with a color or other optical property misalignment within the range encompassed by bar 715 may be from an object or portion of an object that has been obscured in the image, for example, by decorative decoration or paint coating.

[0090] Figure 8 is a flowchart of a computer-based process 800 for annotating landmarks appearing in a 3D model, such as 3D model 425. Process 800 may be performed by, for example, a machine-readable instruction set, a hardware assembly, or one or more data processing devices that perform data processing activities according to the logic of these instructions and / or combinations of instructions. Process 800 may be performed independently or in combination with other activities. For example, process 800 may be performed in combination with process 300 (Figure 3).

[0091] In step 805, the system performing step 800 renders a collection of substitute images of the object using a 3D model of the object formed from its component parts. The substitute images are not actual images of the real-world object; rather, they are substitutes for images of the real-world object. These substitute images generally show the object from various different angles, orientations, and / or dimensions—as if the camera were imaging various different relative-pose objects—with the same dimensionality as the image received in step 305 in step 300 (Figure 3).

[0092] Substitute images can be rendered using 3D models in many ways. For example, ray tracing or other computer graphics techniques may be used. Generally, 3D models of objects are perturbed to render substitute images. Therefore, different substitute images may represent different variations of the 3D model. Generally, perturbation can mimic real-world variations in an object—or part of an object—represented by a 3D model. For example, in a 3D model of a car, the colors of the exterior paint and interior trim may be perturbed. In some examples, parts (features such as tires, wheel caps, and roof racks) may be added, removed, or replaced. As another example, in a 3D model of an organ, physiologically relevant size and relative size variations may be used to perturb the 3D model.

[0093] In some embodiments, aspects other than the 3D model may be varied to further alter the substitute image. Generally, the variations are, for example, - Variations in imaging devices (e.g., camera resolution, zoom, focus, aperture speed), - Variations in image processing (e.g., digital data compression, chroma subsampling), and, - Variations in imaging conditions (e.g., lighting, weather, background color, and shape) It can mimic variations of the real world, including [specific examples of real-world phenomena].

[0094] In some embodiments, the surrogate image is rendered in a reference coordinate system. This reference coordinate system may include background features that appear behind the object, and foreground features that appear in front of the object—and potentially obscure parts of the object. Generally, the reference coordinate system reflects the real-world environment in which the object might be observed. For example, a car might be rendered in a reference coordinate system similar to a parking lot, while organs might be rendered in a physiologically relevant context. The reference coordinate system can be further modified to further alter the two-dimensional image.

[0095] Generally, it is desirable for the substitute images to vary significantly. Furthermore, the number of substitute images—and the degree of variation—may depend on the complexity of the object and the image processing ultimately performed using the landmarks annotated in the 3D model. As an example, more than 2000 very different substitute images (through relative posing and substitution) of a car may be rendered. Since the substitute images are rendered from the 3D model, complete knowledge information about the object's position in the substitute images can be retained regardless of the number and degree of variation of the substitute images.

[0096] In step 810, the system performing step 800 assigns each region of the object shown in the surrogate image to a part of the object. As described above, the 3D model of an object can be divided into identifiable component parts based on function and / or structure. When the surrogate image of the 3D model is rendered, the part to which each region in the image belongs can be preserved. —This may be a pixel or other area in a two-dimensional image—the region can therefore be assigned to the corresponding component part of the 3D model using complete knowledge information derived from the 3D model.

[0097] In step 815, the system performing step 800 identifies identifiable regions of the part in the surrogate image. Identifiable regions of a part are areas (e.g., pixels or groups of pixels) that can be identified in the surrogate image using one or more image processing techniques. For example, in some embodiments, the corners of regions in each image assigned to the same part are detected using, for example, a Moravec corner detector or a Harris corner detector (https: / / en.wikipedia.org / wiki / Harris_Corner_Detector). As another example, an image feature detection algorithm, for example SIFT / SURF / HOG / (https: / / en.wikipedia.org / wiki / Scale-invariant_feature_transform), may be used to define identifiable regions.

[0098] In step 820, the system performing step 800 identifies a collection of landmarks in the 3D model by projecting the identifiable regions in the substitute image back onto the 3D model. The volumes in the 3D model corresponding to the identifiable regions in the substitute image are identified as landmarks in the 3D model.

[0099] In some embodiments, one or more filtering techniques may be applied before or after back projection onto the 3D model to reduce the number of these landmarks and to ensure quality. For example, in some embodiments, areas close to the outer boundaries of an object in the surrogate image may be discarded before back projection. As another example, back projections of areas too far from the corresponding part in the 3D model may be discarded.

[0100] In some embodiments, only volumes in a 3D model that meet a threshold criterion are identified as landmarks. The threshold criterion can be identified by many methods. For example, candidate landmarks in a 3D model, volumes identified by back projection from different surrogate images rendered using different relative poses and variations, may be collected. Clusters of candidate landmarks may be identified, and anomalous candidate landmarks may be discarded. For example, clustering techniques, such as the OPTICS algorithm (https: / / en.wikipedia.org / wiki / OPTICS_algorithm, a variation of DBSCAN, https: / / en.wikipedia.org / wiki / DBSCAN), may be used to identify clusters of candidate landmarks. The effectiveness of clustering may be evaluated using, for example, the Calinski-Harabasz index (i.e., variance ratio criterion) or other criteria. In some embodiments, clustering techniques may be selected and / or tuned to improve the effectiveness of clustering (e.g., by tuning the hyperparameters of the clustering algorithm). If necessary, candidate landmarks within a cluster that are closer to each other than the threshold criterion may be merged. In some embodiments, candidate landmark clusters located in different parts of a 3D model may be further merged into a single cluster. In some embodiments, the centroids of several candidate landmarks within a cluster may be represented as a single landmark.

[0101] In some embodiments, landmarks in a 3D model may be filtered based on the predictability of their positions or other characteristics in a surrogate image rendered from the 3D model. For example, if the position of a 3D landmark in the surrogate image is too difficult to predict (e.g., predicted inaccurately by more than a time threshold percentage, or predicted with insufficient accuracy), that 3D landmark may be discarded. As a result, only 3D landmarks with positions in the surrogate image that can be relatively easily predicted by the landmark prediction unit remain.

[0102] In some cases, the number of landmarks identified may be adjusted to suit a particular data processing activity. For example, the number of landmarks may be: - In 805, rendering more or fewer substitute images using more or fewer replacements of 3D models, - In 810, dividing a 3D model into more or fewer parts to which a region is assigned. - In 815, relax or tighten the constraints for considering a region as identifiable, and / or - After 820, relax or tighten the constraints for filtering landmarks after back-projecting identifiable regions onto the 3D model. It can be adjusted by many methods, including [mention specific methods here].

[0103] The subjects and embodiments of operation described herein may be implemented in digital electronic circuits, including structures disclosed herein and their structural equivalents, or in computer software, firmware, or hardware, or in one or more combinations thereof. Embodiments of the subjects described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded in a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiving device for execution by a data processing device. The computer storage medium may be, or may be, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or one or more combinations thereof. Furthermore, although the computer storage medium is not a propagating signal, the computer storage medium may be a source or destination for computer program instructions encoded in an artificially generated propagating signal. Computer storage media can also be, or be included in, one or more independent physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0104] The operations described herein may be implemented as operations performed by a data processing device on data stored in one or more computer-readable storage devices or received from other sources.

[0105] The term "data processing device" encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, systems on a chip, or a combination of these. A device may include purpose-specific logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, a device may further include code that generates the execution environment for the computer program in question, such as processor firmware, protocol stacks, database management systems, operating systems, cross-platform execution environments, virtual machines, or code comprising one or more of these. Devices and execution environments can realize various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0106] A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it may be deployed in any form, including as a standalone program or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may, but is not required, correspond to a file in a file system. A program may be stored in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file assigned to the program in question, or in multiple collaborative files (e.g., files storing one or more modules, subprograms, or parts of code). A computer program may be deployed to run on one computer, or on multiple computers located in one place or distributed across multiple locations and interconnected by a communication network.

[0107] The processes and logic flows described herein may be implemented by one or more programmable processors that execute one or more computer programs to perform operations by processing input data and generating outputs. The processes and logic flows may be further implemented by purpose-specific logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the apparatus may be implemented as purpose-specific logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits).

[0108] Processors suitable for executing computer programs include, as examples, general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors in any type of digital computer. Generally, a processor receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a processor for performing actions according to instructions, and one or more memory devices for storing instructions and data. Generally, a computer further includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is coupled operable to receive data from or transmit data to or both of these mass storage devices. However, a computer does not necessarily have such devices. Furthermore, a computer may be incorporated into another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive). Devices suitable for storing computer program instructions and data include, as examples, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks; encompassing all forms of non-volatile memory, media, and memory devices. Processors and memory may be complemented by or incorporated into purpose-specific logic circuits.

[0109] To provide user interaction, embodiments of the subject matter described herein may be implemented in a computer that includes a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user, and a pointing device, such as a keyboard and a mouse or trackball, to which the user can provide input to the computer. Other types of devices may also be used to provide user interaction, for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, verbal, or haptic input. In addition, the computer may interact with the user by sending documents to and receiving documents from devices used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0110] Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the claims set forth below. In some examples, the actions described in the claims may be performed in a different order and still yield the desired results. In addition, the steps depicted in the accompanying drawings do not necessarily require a specific order or sequence shown to obtain the desired results. In certain embodiments, multitasking and parallel processing may be beneficial.

[0111] Many embodiments are described. Nevertheless, it is understood that various modifications are possible. Therefore, other embodiments are within the scope of the claims described below. (Aspect 1) A method for image processing, wherein the method is Receiving an image of an object instance and a three-dimensional model of the said object, To detect a first set of landmarks of the instance of the object in a two-dimensional image, Estimating the pose of the instance of the object in the received image with respect to the imaging device that acquired the image, wherein the relative pose in the received image is estimated from the first plurality of detected landmarks, Using the estimated relative pose, project landmarks from the three-dimensional model of the object onto the dimensional space of the received image of the instance of the object, Comparing the characteristics of the corresponding projected landmark and the first plurality of detected landmarks in the dimensional space, To determine whether a threshold-level positional correspondence exists between the corresponding projected landmark and the first plurality of detected landmarks and their positions, Methods that include... (Aspect 2) The method described above is The detection of multiple landmarks of the instance of the object in the image, wherein the detected multiple instances include more landmarks than the first multiple landmarks. Selecting the first set of landmarks from among the detected set of landmarks, The method according to embodiment 1, further comprising: (Aspect 3) The selection of the aforementioned first group of items is random. The method described in Embodiment 2. (Aspect 4) The selection of the first set of landmarks is guided by the characteristics of the landmarks. The method described in Embodiment 2. (Aspect 5) The aforementioned characteristic is how certain it is that a given landmark has been properly detected. The method according to aspect 4. (Aspect 6) In response to the determination that no positional correspondence exists at the threshold level, Re-estimating the relative pose of the instance of the object in the first l image from a second plurality of landmarks of the instance of the object detected in the first l image, wherein at least some of the landmarks among the second plurality of differ from the landmarks among the first plurality of, Using the re-estimated relative pose, the landmarks from the three-dimensional model of the object are projected onto the dimensional space of the received image of the instance of the object. Comparing the features of the corresponding projected landmark and the second plurality of detected landmarks in the dimensional space, To determine whether a positional correspondence relationship of the threshold level exists between the positions of the corresponding projected landmark and the second plurality of detected landmarks, The method according to embodiment 1, further comprising: (Aspect 7) The process further includes selecting the second plurality of landmarks of the instance of the object, and selecting the second plurality of Identifying the corresponding projected landmark and the landmark among the first plurality of them, with a relatively large positional difference, Excluding the landmarks from the first group of items that differ relatively significantly from the second group of items, The method according to embodiment 6, including the method described in embodiment 6. (Pattern 8) The process further includes selecting the second plurality of landmarks of the instance of the object, and selecting the second plurality of Identifying the positions of the corresponding projected landmark and the two-dimensional landmark among the first plurality, which involve a relatively large positional difference, Removing landmarks from the second plurality of objects that are in the vicinity of the corresponding projected landmark and the detected landmark, which have a relatively large positional difference, The method according to embodiment 6, including the method described in embodiment 6. (Aspect 9) The process further includes selecting the second plurality of landmarks of the instance of the object, and selecting the second plurality of Identifying the direction of the positional misalignment between the corresponding projected landmark and the detected landmark, Removing from the second plurality of landmarks a landmark among the first plurality that has a positional displacement direction different from the positional displacement direction of most of the positional displacement between the corresponding projected landmark and the detected landmark, The method according to embodiment 6, including the method described in embodiment 6. (Aspect 10) In response to the determination that a positional correspondence relationship exists at the threshold level, Identifying a subset of the landmarks of the instances of the object detected in the image, which have a relatively large deviation from the corresponding projected landmark, To draw conclusions about the instance of the object based on the subset of the detected landmarks, The method according to embodiment 1, further comprising: (Aspect 11) Reaching the aforementioned conclusion includes specifying the portion of the instance of the object that is deformed or damaged, The method according to embodiment 10. (Aspect 12) Deriving the above conclusion involves quantifying the magnitude of a relatively large displacement, the direction of the relatively large displacement, or both the magnitude and the direction. The method according to embodiment 10. (Aspect 13) Determining whether or not the aforementioned positional correspondence relationship of threshold levels exists is, Combining the positional differences between multiple corresponding projected landmarks and detected landmarks, Comparing the aforementioned combination of positional differences with threshold conditions, The method according to embodiment 1, including the method described in embodiment 1. (Aspect 14) Estimating the relative pose of the instance of the object To form a first estimation result of the relative pose of the instance of the object in the received image, To identify that the quality of the first relative pose estimation result is insufficient, In response, a second estimation result of the relative pose of the instance of the object in the received image is formed, The method according to embodiment 1, including the method described in embodiment 1. (Aspect 15) Receiving a second image of the instance of the object, To detect a second set of landmarks of the instance of the object in the second image acquired by the second imaging device, Estimating the pose of the instance of the object in the second image with respect to the second imaging device that acquired the second image, wherein the relative pose in the second image is estimated from a second plurality of detected landmarks, Using the estimated relative pose of the instance of the object and the estimated relative pose of the instance of the object in the second image, the landmark from the three-dimensional model of the object is projected onto the dimensional space of the second image. The method according to embodiment 1, further comprising: (Aspect 16) handle, a) The landmark projected using the estimated relative pose of the instance of the object in the second image, b) The second plurality of detected landmarks, Comparing the characteristics in the aforementioned dimensional space, To determine whether a threshold level correspondence exists between the compared features of the corresponding projected landmark and the first plurality of detected landmarks, The method according to embodiment 15, further comprising: (Aspect 17) The image of the instance is a two-dimensional image. The detected landmark is a two-dimensional landmark, The dimensional space of the received image is a two-dimensional space. The features of the corresponding projected landmark being compared with the second plurality of detected landmarks are positional features in two-dimensional space. The method described in Embodiment 1. (Aspect 18) To detect the landmark of the instance of the object in the image of the instance of the object, Estimating the pose of the instance of the object with respect to the imaging device that acquired the image, Using the estimated pose, project landmarks from the three-dimensional model of the object onto the dimensional space of the image of the instance of the object, Identifying the corresponding projected landmark and the detected landmark, In order to identify the shift of the corresponding landmark, the characteristics of the corresponding projected landmark and the detected landmark are compared in the dimensional space, Identifying a subset of the corresponding landmarks that exhibit a deviation greater than the threshold condition, Based on the identified subset, identify the deformation, movement, or obscurity of the instance of the object or a portion of the instance of the object, Methods that include... (Aspect 19) Identifying the deformation, movement, or obscurity includes identifying the damaged portion of the instance of the object. The method described in Embodiment 18. (Aspect 20) Identifying the deformation, movement, or ambiguity To detect the landmark of the instance of the object in a second image of the instance of the object, Based on the location of the landmark detected in the second image, the deformation, movement, or blurring is identified. The method according to embodiment 18, including the method described in embodiment 18. (Aspect 21) Identifying clusters of spatially close landmarks of the subset of the corresponding landmarks that are misaligned, Based on the cluster of spatially close landmarks, the deformation or movement of the object is identified. The method according to embodiment 18, further comprising: (Aspect 22) Identifying the corresponding landmark includes comparing the contexts of the projected and detected landmarks in order to identify the corresponding landmark. The method described in Embodiment 18. (Aspect 23) The image of the instance is a two-dimensional image. The detected landmark is a two-dimensional landmark, The dimensional space of the received image is a two-dimensional space. The feature compared to identify the discrepancy is a two-dimensional position feature. The method described in Embodiment 18. (Additional note 1) A method for image processing, wherein the method is Receiving an image of the physical instance of an object and a three-dimensional model of the said object, To detect a first set of landmarks of the physical instance of the object in a two-dimensional image, Estimating the pose of the physical instance of the object in the received image with respect to the imaging device that acquired the image, wherein the relative pose in the received image is estimated from the first plurality of detected landmarks. Using the estimated relative pose, the landmarks from the three-dimensional model of the object are projected onto the dimensional space of the received image of the physical instance of the object. Comparing the features of the corresponding projected landmark with the first plurality of detected landmarks in the dimensional space, To determine whether a threshold-level positional correspondence exists between the positions of the corresponding projected landmark and the first plurality of detected landmarks, Includes, In response to the determination that the above method does not have a positional correspondence relationship for the threshold level, Reestimating the relative pose of the physical instance of the object in the first image from a second set of landmarks of the physical instance of the object detected in the first image, wherein at least some of the landmarks among the second set of landmarks are different from the landmarks among the first set of landmarks. Using the re-estimated relative pose, the landmarks from the three-dimensional model of the object are projected onto the dimensional space of the received image of the physical instance of the object. Comparing the features of the corresponding projected landmark and the second plurality of detected landmarks in the dimensional space, To determine whether a positional correspondence relationship of the threshold level exists between the positions of the corresponding projected landmark and the second plurality of detected landmarks, including, method. (Additional note 2) The method described above is The detection of multiple landmarks of the physical instance of the object in the image, wherein the detected multiple include more landmarks than the first multiple landmarks. Selecting the first set of landmarks from among the detected set of landmarks, The method described in Appendix 1, further including the following: (Additional note 3) The selection of the aforementioned first group of items is random. The method described in Appendix 2. (Additional note 4) The selection of the first set of landmarks is guided by a landmark-by-landmark comparison of the features of the corresponding projected landmark and other detected landmarks. The method described in Appendix 2. (Additional note 5) The characteristic is how certain it is that a given landmark has been properly detected. The method described in Appendix 4. (Additional note 6) The process further includes selecting the second plurality of landmarks of the physical instance of the object, and the selection of the second plurality of Identifying the corresponding projected landmark and the landmark among the first plurality of them, with a relatively large positional difference, To remove the landmark from the first plurality of which have a relatively large positional difference from the second plurality of, The method described in Appendix 1, including the method described in Appendix 1. (Additional note 7) The process further includes selecting the second plurality of landmarks of the physical instance of the object, and the selection of the second plurality of Identifying the positions of the corresponding projected landmark and the two-dimensional landmark among the first plurality, which involve a relatively large positional difference, Removing landmarks from the second plurality of objects that are in the vicinity of the corresponding projected landmark and the detected landmark, which have a relatively large positional difference, The method described in Appendix 1, including the method described in Appendix 1. (Additional note 8) The process further includes selecting the second plurality of landmarks of the physical instance of the object, and the selection of the second plurality of Identifying the direction of the positional misalignment between the corresponding projected landmark and the detected landmark, Removing from the second plurality of landmarks a landmark among the first plurality that has a positional displacement direction different from the positional displacement direction of most of the positional displacement between the corresponding projected landmark and the detected landmark, The method described in Appendix 1, including the method described in Appendix 1. (Additional note 9) In response to the determination that a positional correspondence relationship exists at the threshold level, Identifying a subset of the landmarks of the physical instances of the object detected in the image, which have a relatively large deviation from the corresponding projected landmark, To draw conclusions about the physical instance of the object based on the subset of the detected landmarks, The method described in Appendix 1, further including the following: (Additional note 10) Reaching the aforementioned conclusion involves specifying the portion of the physical instance of the object that is deformed or damaged, The method described in Appendix 9. (Additional note 11) Deriving the above conclusion involves quantifying the magnitude of a relatively large displacement, the direction of the relatively large displacement, or both the magnitude and the direction. The method described in Appendix 9. (Additional note 12) Determining whether or not the aforementioned positional correspondence relationship of threshold levels exists is, Combining the positional differences between multiple corresponding projected landmarks and detected landmarks, Comparing the aforementioned combination of positional differences with threshold conditions, The method described in Appendix 1, including the method described in Appendix 1. (Additional note 13) To estimate the relative pose of the physical instance of the object, To form a first estimation result of the relative pose of the physical instance of the object in the received image, To identify that the quality of the first relative pose estimation result is insufficient, In response, a second estimation result of the relative pose of the physical instance of the object in the received image is formed, The method described in Appendix 1, including the method described in Appendix 1. (Additional note 14) Receiving a second image of the physical instance of the object, To detect a second set of landmarks of the physical instance of the object in the second image acquired by the second imaging device, Estimating the pose of the physical instance of the object in the second image with respect to the second imaging device that acquired the second image, wherein the relative pose in the second image is estimated from a second plurality of the detected landmarks, Using the estimated relative pose of the physical instance of the object and the estimated relative pose of the physical instance of the object in the second image, the landmark from the three-dimensional model of the object is projected onto the dimensional space of the second image. The method described in Appendix 1, further including the following: (Additional note 15) handle, a) The landmark projected using the estimated relative pose of the physical instance of the object in the second image, b) The second plurality of the detected landmarks, Comparing the characteristics in the aforementioned dimensional space, To determine whether a threshold level correspondence exists between the features of the corresponding projected landmark and the first plurality of detected landmarks, The method described in Appendix 14, further including the method described in Appendix 14. (Additional note 16) The image of the physical instance is a two-dimensional image. The detected landmark is a two-dimensional landmark, The dimensional space of the received image is a two-dimensional space. The features of the corresponding projected landmark being compared with the second plurality of detected landmarks are positional features in two-dimensional space. The method described in Appendix 1. (Additional note 17) To detect the landmark of the physical instance of the object in an image of the physical instance of the object, Estimating the pose of the physical instance of the object with respect to the imaging device that acquired the image, Using the estimated pose, project landmarks from the three-dimensional model of the object onto the dimensional space of the image of the physical instance of the object, Identifying the corresponding projected landmark and the detected landmark, In order to identify the shift of the corresponding landmark, the characteristics of the corresponding projected landmark and the detected landmark are compared in the dimensional space, Identifying a subset of the corresponding landmarks that exhibit a deviation greater than the threshold condition, Based on the identified subset, identify deformation, movement, or obscurity of the physical instance of the object or a portion of the physical instance of the object, Methods that include... (Additional note 18) Identifying the deformation, movement, or obscurity includes identifying the damaged portion of the physical instance of the object. The method described in Appendix 17. (Additional note 19) Identifying the deformation, movement, or ambiguity To detect landmarks of the physical instance of the object in a second image of the physical instance of the object, Based on the location of the landmark detected in the second image, the deformation, movement, or blurring is identified. The method described in Appendix 17, including the method described in Appendix 17. (Additional note 20) Identifying clusters of spatially close landmarks of the subset of the corresponding landmarks that are misaligned, Based on the cluster of spatially close landmarks, the deformation or movement of the object is identified. The method described in Appendix 17, further including the method described in Appendix 17. (Additional note 21) Identifying the corresponding landmark includes comparing the contexts of the projected and detected landmarks in order to identify the corresponding landmark. The method described in Appendix 17. (Additional note 22) The image of the physical instance is a two-dimensional image. The detected landmark is a two-dimensional landmark, The dimensional space of the received image is a two-dimensional space. The feature compared to identify the discrepancy is a two-dimensional position feature. The method described in Appendix 17.

Claims

1. A method for image processing, wherein the method is Receiving an image of the physical instance of an object and a three-dimensional model of the said object, To detect a first set of landmarks of the physical instance of the object in the received image, Estimating the relative pose of the physical instance of the object in the received image with respect to the imaging device that acquired the received image, wherein the relative pose in the received image is estimated from a first plurality of detected landmarks, Using the estimated relative pose, project landmarks from the three-dimensional model of the object onto the dimensional space of the received image of the physical instance of the object, Comparing the characteristics of the corresponding projected landmark with the first plurality of detected landmarks in the dimensional space, To determine whether a threshold-level positional correspondence exists between the positions of the corresponding projected landmark and the first plurality of detected landmarks, Includes, In response to the determination that the above method does not have a positional correspondence relationship for the threshold level, Reestimating the relative pose of the physical instance of the object in the first image from a second plurality of landmarks of the physical instance of the object detected in the first image, wherein at least some of the landmarks among the second plurality are different from the landmarks among the first plurality. Using the re-estimated relative pose, the landmarks from the three-dimensional model of the object are projected onto the dimensional space of the received image of the physical instance of the object. Comparing the characteristics of the corresponding projected landmark and the second plurality of detected landmarks in the dimensional space, To determine whether a positional correspondence relationship of the threshold level exists between the positions of the corresponding projected landmark and the second plurality of detected landmarks, Includes, In response to the determination that the above method has found that a positional correspondence relationship exists at the threshold level, Identifying a subset of the landmarks of the physical instances of the object detected in the image, which have a relatively large deviation from the corresponding projected landmark, To draw conclusions about the physical instance of the object based on the subset of the detected landmarks, This also includes, method.

2. The method described above is To detect multiple landmarks of the physical instance of the object in the received image, wherein the detected multiple include more landmarks than the first multiple landmarks. Selecting the first set of landmarks from among the detected set of landmarks, The method according to claim 1, further comprising:

3. The selection of the first plurality of detected landmarks is random. The method according to claim 2.

4. The selection of the first set of landmarks is guided by a landmark-by-landmark comparison of the features of the corresponding projected landmark and other detected landmarks. The method according to claim 2.

5. The characteristic is how certain it is that a given landmark has been properly detected. The method according to claim 4.

6. The process further includes selecting the second plurality of landmarks of the physical instance of the object, and the selection of the second plurality of Identifying the corresponding projected landmark and the landmark among the first plurality of landmarks, with a relatively large positional difference, To remove the landmark from the first plurality of which have a relatively large positional difference from the second plurality of, The method according to claim 1, including the method described in claim 1.

7. The process further includes selecting the second plurality of landmarks of the physical instance of the object, and the selection of the second plurality of Identifying the positions of the corresponding projected landmark and the two-dimensional landmark among the first plurality, which involve a relatively large positional difference, To remove a landmark from the second plurality of objects that is in the vicinity of the corresponding projected landmark and the detected landmark, which have a relatively large positional difference, The method according to claim 1, including the method described in claim 1.

8. The process further includes selecting the second plurality of landmarks of the physical instance of the object, and the selection of the second plurality of Identifying the direction of the positional misalignment between the corresponding projected landmark and the detected landmark, To exclude from the second plurality of landmarks a landmark among the first plurality of landmarks whose displacement direction is different from the displacement direction of most of the displacement between the corresponding projected landmark and the detected landmark, The method according to claim 1, including the method described in claim 1.

9. Reaching the aforementioned conclusion involves specifying the portion of the physical instance of the object that is deformed or damaged, The method according to claim 1.

10. Deriving the above conclusion involves quantifying the magnitude of a relatively large displacement, the direction of the relatively large displacement, or both the magnitude and the direction. The method according to claim 1.

11. Determining whether or not the aforementioned positional correspondence relationship of threshold levels exists is, Combining the positional differences between multiple corresponding projected landmarks and detected landmarks, Comparing the aforementioned combination of positional differences with threshold conditions, The method according to claim 1, including the method described in claim 1.

12. To estimate the relative pose of the physical instance of the object, To form a first estimation result of the relative pose of the physical instance of the object in the received image, To identify that the quality of the first relative pose estimation result is insufficient, In response, a second estimation result of the relative pose of the physical instance of the object in the received image is formed, The method according to claim 1, including the method described in claim 1.

13. Receiving a second image of the physical instance of the object, To detect a second set of landmarks of the physical instance of the object in the second image acquired by the second imaging device, Estimating the second image-relative pose of the physical instance of the object in the second image with respect to the second imaging device that acquired the second image, wherein the second image-relative pose in the second image is estimated from a second plurality of landmarks detected in the second image. Using the estimated second image relative pose of the physical instance of the object in the second image, the landmarks from the three-dimensional model of the object are projected onto the dimensional space of the second image, The method according to claim 1, further comprising:

14. a) Features in each of at least some of the landmarks projected using the estimated second image relative pose of the physical instance of the object in the second image, b) The features of the corresponding landmark among the second plurality of landmarks detected in the second image, Comparing in the aforementioned dimensional space, To determine whether a threshold-level correspondence exists between the features of the corresponding projected landmark and the second set of detected landmarks in the second image, The method according to claim 13, further comprising:

15. The image of the physical instance is a two-dimensional image. The detected landmark is a two-dimensional landmark, The dimensional space of the received image is a two-dimensional space. The features of the corresponding projected landmark being compared with the second plurality of detected landmarks are positional features in two-dimensional space. The method according to claim 1.

Images