Annotating 3D Models with Visible Signs of Use in 2D Images

JP2024506691A5Active Publication Date: 2024-11-27アイエヌエイアイティ エスエイ
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023548937
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-03-03
Filing Date
2022-01-26
Publication Date
2024-11-27
Estimated Expiration
2042-01-26

AI Technical Summary

Technical Problem

Existing 3D models lack accurate representation of real-world usage evidence, such as wear and tear or damage, which can affect simulation and evaluation accuracy.

Method used

A method to annotate 3D models using visible usage evidence from 2D images by projecting and matching signs of use from wide and narrow field of view images to accurately determine and correct the positioning of these signs on the 3D model.

Benefits of technology

Enhances the accuracy of 3D models by incorporating real-world usage evidence, improving simulation results and enabling precise remedial actions or failure estimation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

A method, system, and apparatus, including a computer program encoded in a computer storage medium, for annotating a 3D model with evidence of use visible in a 2D image. In one embodiment, the method is implemented by a data processing device. The method may include projecting evidence of use in a relatively wide field of view image of an instance of an object onto a 3D model of the object based on a pose of the instance in the relatively wide field of view image, and estimating a relative pose of the instance of the object in the relatively narrow field of view image based on a matching relationship between the evidence of use in the relatively wide field of view image and the same evidence of use in the relatively narrow field of view image.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to Greek Application No. 20210100106, filed February 18, 2021, and U.S. Application No. 17 / 190,646, filed March 3, 2021, the contents of which are incorporated herein by reference.

[0002] This document relates to annotating 3D models with visible evidence of use in 2D images. [Background technology]

[0003] Many man-made objects are virtually designed in computers with the aid of 3D models. Examples include automobiles, aircraft, buildings, consumer products, and their constituent components. 3D models generally contain detailed information about the size, shape, and orientation of the structural features of the object. Some 3D models also contain information about other features, including, for example, composition, material properties, electrical properties, etc. In addition to being used for design, 3D models can also be used for testing and other purposes.

[0004] Regardless of the object being modeled and the use of the model, the model is often an idealized abstraction of a real-world instance of the object. For example, a real-world instance of an object often includes signs of use that are not captured in the model. Examples of signs of use include normal wear and tear that occurs over time as well as damage caused by a separate incident. Signs of damage include deformations, scratches, and dents on a vehicle or structure. In any case, signs of use of a real-world instance of an object rarely match a 3D model of the object - or even signs of use of another real-world instance of the same object. Summary of the Invention [Problem to be solved by the invention]

[0005] This specification describes techniques related to annotating 3D models using evidence of use. Annotating a 3D model adds structured information to the model. In this case, the structured information is related to the evidence of use. In some examples, the structured information related to the evidence of use can be used, for example, to transform or otherwise modify a 3D model of an object so that it fits a particular instance of the object. The modified 3D model can be used in various simulation and evaluation processes.

[0006] The signs of use are imaged in 2D images, e.g., actual images captured by a smartphone or other common imaging device. In general, a 3D model can be adequately annotated from a 2D image, provided that certain minimum requirements are met by the 2D image, e.g., sufficient resolution of the signs of use and an appropriate perspective for the signs of use. Furthermore, these minimum requirements are generally intuitive for a human user, who is adept at recognizing object features in the user's daily life. [Means for solving the problem]

[0007] In general, one innovative aspect of the subject matter described herein may be embodied by a method implemented by a data processing device, which may include projecting traces of use in the relatively wide field of view image of an instance of an object onto a 3D model of the object based on a pose of the instance in the relatively wide field of view image, and estimating a relative pose of the instance of the object in the relatively narrow field of view image based on a matching relationship between the traces of use in the relatively wide field of view image and the same traces of use in the relatively narrow field of view image.

[0008] This and other embodiments may include one or more of the following features: The method may include computing virtual positions of the evidence of use in the relatively narrow field of view images using the estimated pose and projections of the evidence of use onto the 3D model, comparing the virtual positions with actual positions of the evidence of use in the relatively narrow field of view images, and identifying a subset of the evidence of use that is improperly projected onto the 3D model based on the comparison. The 3D model may be deformed using a second subset of the evidence of use that is properly projected onto the 3D model.

[0009] The method may include projecting the improperly projected ones of the indications of use onto a 3D model of the object based on a relative pose of the instances of the object in the relatively narrow field of view image. Projecting the improperly projected ones of the indications of use onto the 3D model may include identifying an area of ​​the relatively narrow field of view image that includes a first one of the improperly projected ones of the indications of use, matching the first one of the improperly projected ones of the indications of use with a first one of the indications of use in the relatively wide field of view image, and projecting the first one of the indications of use in the relatively wide field of view image onto the 3D model of the object.

[0010] A subset of the improperly projected indications of use may be identified based on a misalignment between the virtual positions and the actual positions of the indications of use in the relatively narrow field of view image. The method may include filtering the subset of the indications of use that are improperly projected onto the 3D model from the matching relations to establish a proper subset of the matching relations, and re-estimating a relative pose of the instances of the object in the relatively narrow field of view image based on the subset of the matching relations.

[0011] Projecting the evidence of use in the relatively wide field of view image onto the 3D model may include identifying a pose in the relatively wide field of view image. The method may include identifying a dominant color of the instance of the object in the relatively narrow field of view image, identifying regions in the relatively wide field of view image, in the relatively narrow field of view image, or in both the relatively wide field of view image and the relatively narrow field of view image that are offset from the dominant color, and matching the identified regions to match the evidence of use in the relatively wide field of view image with the evidence of use in the relatively narrow field of view image.

[0012] The method may include identifying deviations from an ideal state for instances of an object in the relatively narrow field of view images and matching the deviations in the relatively narrow field of view images to the relatively wide field of view images to match the signs of use in the relatively narrow field of view images. The method may include estimating relative poses of instances of the object in a plurality of relatively narrow field of view images and computing a virtual position of the same signs of use in the relatively narrow field of view images using the estimated poses.

[0013] In another embodiment, the subject matter described herein may be embodied in a method implemented by a data processing device, the method may include annotating a 3D model of an object using usage evidence from two or more 2D images of an instance of the object. Annotating the 3D model may include receiving a 3D model and 2D images, where a first of the 2D images is a relatively wide field of view image of the instance and a second of the 2D images is a relatively narrow field of view image of the instance; matching visible signs of use in the relatively wide field of view image and the relatively narrow field of view image; projecting the signs of use in the relatively wide field of view image onto a 3D model of the object; estimating a pose of the instance in the relatively narrow field of view image using the projection of the signs of use onto the 3D model and the matched signs of use in the relatively wide field of view image and the relatively narrow field of view image; computing virtual positions of the signs of use in the relatively narrow field of view image using the estimated pose and the projection of the signs of use onto the 3D model; comparing the virtual positions with actual positions of the signs of use in the relatively narrow field of view image to identify improperly projected ones of the signs of use; and removing the improperly projected ones of the signs of use from the projection onto the 3D model of the object.

[0014] This and other embodiments may include one or more of the following features. The method may include projecting improperly projected ones of the signs of use onto a 3D model of the object based on a relative pose of the instances of the object in the relatively narrow field of view image. Projecting the improperly projected ones of the signs of use onto the 3D model may include identifying an area of ​​the relatively narrow field of view image that includes a first one of the improperly projected ones of the signs of use, matching the first one of the improperly projected ones of the signs of use with a first one of the signs of use in the relatively wide field of view image, and projecting the first one of the signs of use in the relatively wide field of view image onto the 3D model of the object.

[0015] A subset of improperly projected evidence of use may be identified based on misalignment between the virtual location and the actual location of the evidence of use in the relatively narrow field of view image. Matching the evidence of use may include identifying a dominant color of the instance of the object in the relatively narrow field of view image and identifying areas in the relatively wide field of view image, the relatively narrow field of view image, or both the relatively wide field of view image and the relatively narrow field of view image that deviate from the dominant color. Matching the evidence of use may include identifying deviations from an ideal state of the instance of the object in the relatively narrow field of view image and matching the deviations in the relatively narrow field of view image to the relatively wide field of view image. The method may include deforming the 3D model using properly projected evidence of use from a projection of the object onto the 3D model.

[0016] Other embodiments of the above-described methods include corresponding systems and devices configured to perform the actions of the methods, and computer storage media encoded with a computer program, the program including instructions that, when executed by a data processing device, cause the data processing device to perform the actions of the methods.

[0017] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a schematic diagram of the acquisition of a collection of different images of an object. [Diagram 2] FIG. 2 is a schematic diagram of a collection of two-dimensional images acquired by one or more cameras. [Diagram 3]FIG. 3 is a flowchart of a computer-implemented process for annotating a 3D model with evidence of use from 2D images. [Figure 4] FIG. 4 includes a schematic representation of a computer-implemented process for annotating a 3D model with evidence of usage from 2D images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Like reference numbers and designations in the various drawings indicate similar elements.

[0020] 1 is a schematic diagram of the acquisition of a collection of different images of an object 100. For illustrative purposes, the object 100 is shown as an assembly of idealized unmarked geometric parts (e.g., a cube, a polyhedron, a parallelepiped, etc.). However, in real applications, the object typically has a more complex shape and has texture or is otherwise marked, for example, using decorative decorations, wear marks, or other markings on the underlying shape.

[0021] A collection of one or more imaging devices (shown in this example as cameras 105, 110, 115, 120, 125) may be positioned sequentially or simultaneously at different relative positions around the object 100 and oriented at different relative angles with respect to the object 100. The positions may be distributed in three-dimensional space around the object 100. The orientation may also vary in three dimensions, i.e., all of the Euler angles (or yaw, pitch, and roll) may vary. The relative placement and orientation of the cameras 105, 110, 115, 120, 125 with respect to the object 100 may be referred to as a relative pose. Because the cameras 105, 110, 115, 120, 125 assume different relative poses, each of the cameras 105, 110, 115, 120, 125 acquires a different image of the object 100.

[0022] Even a simplified object, such as object 100, includes many landmarks 130, 131, 132, 133, 134, 135, 136, .... The landmarks are locations of interest in the object 100. The landmarks may be placed at geometric locations in the object or in markings on the underlying geometry. As will be explained in detail below, the landmarks may be used to identify the pose of the object. The landmarks may also be used for other types of image processing, e.g., to classify the object, to extract object features, to identify the location of other structures in the object (geometric structures or markings), to assess damage to the object, and / or to serve as origins from which measurements may be taken in these and other image processing techniques.

[0023] 2 is a schematic diagram of a collection 200 of two-dimensional images acquired by one or more cameras, e.g., cameras 105, 110, 115, 120, 125 (FIG. 1). The images in collection 200 show object 100 at different relative poses. Landmarks, e.g., markers 130, 131, 132, 133, 134, 135, 136, ..., appear in different positions in different images, if they appear at all. For example, in the leftmost image of collection 200, markers 133, 134 are obscured by the rest of object 100. In contrast, in the rightmost image 210, markers 131, 135, 137 are obscured by the rest of object 100.

[0024] As mentioned above, 3D models are often idealized abstractions of real-world instances of objects and do not include, for example, the use evidence present in those real-world instances. However, it is beneficial to include those use evidence in 3D models in various contexts. For example, when a 3D model is used to simulate mechanical or other behavior of a real-world instance of an object, the use evidence may affect the outcome of the simulation. As another example, a 3D model including use evidence may be used to estimate remediation actions to reduce or improve use. For example, a 3D model of a car including damage from an accident may be used to accurately evaluate, for example, the safety or repair costs of the car. As yet another example, a 3D model including use evidence may be used to estimate the time to failure or failure mechanism of an instance of an object. In these and other contexts, annotating a 3D model with use evidence from a 2D image may provide a relatively easy approach to enable a 3D model to include use evidence and to more accurately model a real-world instance.

[0025] 3 is a flowchart of a computer-implemented process 300 for annotating a 3D model with evidence of use from 2D images. Process 300 may be performed by one or more data processing devices performing data processing activities. The activities of process 300 may be performed according to the logic of a set of machine-readable instructions, a hardware assembly, or a combination of these and / or other instructions.

[0026] At 305, a device performing step 300 receives i) a 3D model of a physical object, ii) at least one image of an instance of the object with a relatively wide field of view (FOV), and iii) at least one image of the same instance of the object with a relatively narrow FOV. A 3D model generally represents an object in a three-dimensional space that is generally separated from any reference coordinate system. A 3D model may be generated manually, algorithmically (procedural modeling), or by scanning an actual object. For example, a 3D model may be generated using computer-aided design (CAD) software. Surfaces in the 3D model may be defined using texture mapping.

[0027] Each of the relatively narrow FOVs of an instance of an object includes associated evidence of use that is annotated to a 3D model of the object. Generally, the images of the narrower FOVs show the instance in sufficient detail that--after a pose for the narrower FOV has been identified--the 3D model can be effectively annotated using the evidence of use in the images of the narrower FOVs. Thus, the images of the relatively narrower FOVs can "zoom in" on the associated evidence of use and provide details about the evidence that may be difficult to discern in images with a wider field of view.

[0028] Each of the relatively wide FOV images of the instance shows both relevant evidence of use and other parts of the instance. In some examples, the image of the wider FOV is a full view of the instance, i.e., the entire instance is visible in the image of the wider FOV. In general, the other parts of the instance include at least sufficient features to allow the relative pose of the instance in the image of the wider FOV to be accurately identified. With respect to the evidence of use, at least some of the evidence visible in the image of the narrower FOV is also visible and not hidden in the image of the wider FOV. Although the same evidence of use is visible in both the image of the relatively narrow FOV and the image of the relatively wide FOV, the resolution of the evidence of use in the image of the relatively wide FOV does not need to be high enough to be able to annotate the 3D model based only on the image of the wider FOV. In fact, since the image of the relatively wide FOV necessarily includes a larger portion of the object, the evidence of use is generally shown at too low a resolution for effective annotation of the 3D model.

[0029] With regard to determining the relative pose of the instance in the image with the wider FOV - and thus the correspondence between the features in the wider FOV and the features of the 3D model - this can be done in a number of ways. The extent to which the portions of the object are shown in the image with the wider FOV can affect the approach by which the correspondence between the relative pose and position is determined.

[0030] For example, in some implementations, machine learning models may be used to detect the contours of object instances in images with a wider FOV, in which case it is generally preferable for images with less detail to be a full view of the instance for a given pose.

[0031] As another example, a pose estimation machine learning model may be used to estimate the pose of an instance in an image with a relatively wide FOV. In such a case, it may be acceptable if the image with a relatively wide FOV contains a sufficient number and configuration of landmarks so that pose estimation can be performed. The pose of the object may be estimated based on the detected landmarks, the distances separating them, and their relative positions in the image with the wider FOV. An exemplary machine learning model for landmark detection is detectron2, available at https: / / github.com / facebookresearch / detectron2. An example of a pose estimator that relies on landmark detection is the OpenCV function SolvePNP, described at https: / / docs.opencv.org / master / d7 / d53 / tutorial_py_pose.html.

[0032] As yet another example, in some embodiments, the pose of an instance in an image with a relatively wide FOV may be determined as described in Greek Patent Application No. 20210100068, entitled "ANNOTATION OF TWO-DIMENSIONAL IMAGES," filed on February 2, 2021, the contents of which are incorporated herein by reference.

[0033] As yet another example, in some implementations, the pose of the instance in the image with the relatively wide FOV may be received from an external source along with the image with the relatively wide FOV. For example, the image with the relatively wide FOV may be associated with metadata that specifies the relative placement and orientation of the camera with respect to the instance when the image with the relatively wide FOV was acquired.

[0034] At 310, if necessary, the device performing process 300 determines a pose of an instance of the object in the image of the relatively wide FOV. Exemplary approaches for determining a pose have been described above. As further noted above, in some instances, determination may not be necessary and the pose may be received from an external source.

[0035] At 315, a device performing process 300 matches the evidence of use in the image with the relatively wide FOV to the evidence of use in the image with the relatively narrow FOV. In general, the evidence of use in the two images may be matched pixel by pixel. For example, a feature detection algorithm (e.g., a scale-invariant feature transform) may identify a pixel of interest in one of the images, extract a portion of the image around the pixel, and look for a matching pixel in the other of the images. In some implementations, image registration techniques may also be used - either alone or in combination with a feature detection algorithm.

[0036] In general, the criteria for detecting features or for aligning images are tailored to the context in which the method 300 is performed. For example, when identifying damage caused by an individual event, color deviations may be emphasized, whereas when identifying wear and damage, deviations from an ideal shape (e.g., a uniformly smooth surface or gears with uniform teeth) may be emphasized. As an illustration, assume that scratches and dents on a car are annotated to a 3D model. The dominant color of the image with a relatively narrow FOV may be made to be the color of the car. Pixels may be filtered from the image with the wider FOV, the image with the narrower FOV, or both according to how close the pixel color is to this dominant color. Pixels with large color deviations may be represented as pixels of interest in feature detection algorithms or image alignment techniques.

[0037] In some implementations, the content of the image with a relatively wide FOV can be reduced before matching. For example, the shape of an instance of an object in the image with a relatively wide FOV can be estimated. Pixels that lie outside the boundary of the object can be excluded from including in the matching relationship. For example, pixels that represent objects in the background are excluded, thereby reducing the computational load and the possibility of incorrect matching relationships.

[0038] 4 is a schematic diagram of a computer-implemented process for annotating a 3D model with signs of use from 2D images. In the schematic representation, signs of use in an image 405 with a relatively narrow FOV are matched with signs of use in an image 410 with a relatively wide FOV. In particular, both images 405, 410 show an instance 415 of an object, albeit with different extents and often with different levels of detail. Image 410 is a full view of the instance 415, whereas image 405 is an image with a narrower FOV of signs of use in the instance 415. Although both images 405, 410 contain signs of use, images 405, 410 do not necessarily have to be taken from the same angle. For example, in the illustrated example, the camera that captured image 410 is slightly to the right of the instance 415 compared to the camera that captured image 415.

[0039] Various features in the images 405, 410 are matched, including signs of use and other features, such as corners and edges. The matching relationships are shown diagrammatically as a collection of dashed lines 420. However, because the image 405 has a relatively narrow FOV and shows only a relatively small portion of the instance, the number of features available to be matched other than signs of use is relatively small. Indeed, in many embodiments, only signs of use are matched.

[0040] Returning to FIG. 3, at 320, a device performing process 300 projects the evidence of use from the image with the wider FOV onto a 3D model of the object and identifies 3D coordinates of the evidence of use in the 3D model. The projection depends on the pose of the instance of the object in the image with the wider FOV—determined at 310 or received from an external source. The projection may depend on features in the image with the wider FOV that are not observed in the image with the narrower FOV, e.g., features that are outside the field of view in the image with the narrower FOV. Moreover, because the projection depends on such features, the projection from the image with the wider FOV onto the 3D model is relatively accurate. As a result, the coordinates of the evidence of use can be accurately identified in the 3D model by referencing the positions of these other features.

[0041] 4 further includes a schematic representation of a projection of the evidence of use from the wider FOV image 410 onto the 3D model 425. The projection is shown diagrammatically as a collection of dashed lines 430. As shown, the projection may rely on features in the wider FOV image 410 that are not present in the narrower FOV image 405, including edges and corners that are outside the field of view in the image 405. Because the projection relies on a relatively large portion of the object instance, the projection is relatively accurate. Using the locations of these features as references, the coordinates of the evidence of use may also be precisely identified in the 3D model.

[0042] Returning to Figure 3, at 325, the device performing process 300 uses the coordinates of the usage evidence in the 3D model to estimate a pose of the object instance in the image with the narrower FOV. In particular, the matching relationship between the usage evidence in the image with the narrower FOV and the image with the wider FOV, and the projection of the usage evidence from the image with the wider FOV onto the model, allows the pose of the object instance in the image with the narrower FOV to be estimated. Since images with a narrower FOV generally contain more details about the usage evidence but less details about other features - compared to images with a wider FOV - using the usage evidence to estimate a pose in the image with the narrower FOV takes advantage of the features present in that image.

[0043] At 330, the device implementing 300 uses the estimated pose of the image of the narrower FOV and the coordinates of the traces of use in the 3D model to calculate a virtual location of the traces of use in the image of the narrower FOV. This calculation can be considered as a virtual regeneration of the image of the narrower FOV using the estimated pose and the coordinates of the traces of use in the 3D model as ground truth. In many cases, the entire image of the narrower FOV is not regenerated. Rather, in practice, the calculation identifies where the traces of use are observed in the image of the narrower FOV under the assumption that the estimated pose and the coordinates of the traces of use in the 3D model are corrected.

[0044] At 335, the virtual location of the evidence of use is compared to the actual location of the evidence of use in the image of the narrower FOV. Individual features with insufficient correspondence between the virtual and actual locations may be identified. Since the pose is calculated based on many features, insufficient correspondence indicates that the individual evidence of use is improperly projected onto the 3D model. At 340, such features may be filtered from the matching relationship established at 315.

[0045] For example, a threshold for identifying a poor correspondence may be established based on, for example, an average correspondence of all features, selected features, or both. For example, in some embodiments, an average correspondence of all features may be identified. Features that deviate substantially from this average correspondence of all features may be identified and removed from the matching relationship established at 315.

[0046] As another example, features may be classified according to their uniqueness. A threshold for identifying insufficient correspondence may then be established based on the most unique features. For example, in some implementations, the criteria used to detect features or register images in 315 may be used to identify features that are more unique than other features. Examples include the number of pixels in an area where there is a color shift, or the number of pixels encompassed by deviations from an ideal shape. The more unique feature correspondence may be used to establish a threshold for identifying features with insufficient correspondence. Features that are greater than this threshold may be identified and excluded from the matching relationship established in 315.

[0047] As yet another example, an average correspondence of all features may be identified in a "first pass" through the features. Features that deviate substantially from this average correspondence of all features may be excluded from the recalculation of the correspondence between virtual and actual locations. Other features that deviate substantially from this recalculated correspondence of a subset of features may then be identified and excluded from the matching relationship established in 315.

[0048] Returning to FIG. 4, a pose 435 of the object instance in the narrower FOV image 405 is estimated using the coordinates of the use trace in the 3D model 425. This pose can be used to calculate a virtual position of the use trace in the narrower FOV image. For illustrative purposes, this calculation is represented diagrammatically as a virtual reproduction 440 of the narrower FOV image, but only the position of the use trace in the narrower FOV image needs to be calculated. A comparison 445 between the actual position of the use trace in the narrower FOV image and the calculated virtual position of the use trace is performed to identify a correspondence therebetween. The correspondence is shown diagrammatically as a misregistration, i.e. a collection of differences 450 between the actual two-dimensional position of each feature of the use trace in the less detailed image 405 and the virtual position of the same feature calculated from the relative pose of the narrower FOV image 405 and the projection of the use trace onto the 3D model 425. A threshold 455 for identification and filtering of poor matching relations are further shown diagrammatically. As shown, at least some misalignments 460 are outside of threshold 455. Matching relationships between the locations of signs of use features in narrower FOV image 405 and wider FOV image 410 corresponding to misalignments 460 may be identified and removed from collection 420.

[0049] Returning to FIG. 3, after all unsuitable matching relationships have been removed from the established matching relationships at 315, the remaining matching relationships may be used in a variety of different ways.

[0050] For example, in some instances, few matching relationships are filtered out at 340 and / or a sufficient number of matching relationships have sufficiently high correspondence. Projection of matching relationships with high correspondence onto the 3D model may be used for various downstream purposes including, for example, inferring remedial actions to reduce or improve usage, accurately assessing the safety of an instance, or estimating the time to failure of an instance. In such cases, the illustrated portion of process 300 is effectively terminated, although further downstream actions are envisioned.

[0051] In many instances, a relatively large number of matching relationships are eliminated and / or an insufficient number of matching relationships have a sufficiently high correspondence for the intended downstream purpose, in such cases portions of process 300 may be repeated iteratively until most matching relationships are not eliminated and / or a sufficient number of matching relationships have a sufficiently high correspondence.

[0052] For example, in some implementations, all or part of process 300 may be repeated to look for matching relationships in regions in the image of the narrower FOV from which matching relationships were excluded. These regions may be treated as the image of the narrower FOV of the object instance in the next iteration of process 300. The image of the wider FOV (e.g., image 410 in FIG. 4) from any previous iteration, or the image of the same narrower FOV (e.g., image 405 in FIG. 4) may be used as the image of the wider FOV in this next iteration. In practice, process 300 may be performed multiple times, with each image of the narrower FOV having a progressively smaller field of view and relying on the pose estimation results provided by the previous iteration. As smaller and smaller fields of view are used, the assignment of the trace of use to the 3D model becomes progressively more accurate. In some examples, performance of process 300 may be stopped, for example, when the percentage or number of matching relationships does not increase or increases by less than a threshold amount.

[0053] As another example, the pose estimated at 325 may be used to more accurately match the signs of use in the wider and narrower FOV images. For example, the pose estimated at 325 may be used to identify and discard unrealistic or inappropriate matching relationships at 315 in the next iteration of process 300. As a result, the accuracy of the matching relationships in this next iteration increases, as well as the accuracy of the projection of the signs of use onto the 3D model at 320 and the accuracy of the pose estimation at 325. In some implementations, the threshold for identifying insufficient correspondence between the virtual and actual positions may be made stricter, and process 300 is repeated. In some examples, performance of process 300 may be stopped, for example, when the threshold for identifying insufficient correspondence is no longer strict, or is only strict by a threshold amount.

[0054] As yet another example, process 300 includes: - Increasing accuracy of pose estimation in 325 by identifying and discarding unrealistic or inappropriate matching relationships in 315; - Increasing the percentage or number of matching relationships by performing the process using successive, i.e., relatively narrow FOV, images, each having a smaller field of view; may be repeated for both.

[0055] For example, the accuracy of the pose estimate at 325 may be iteratively improved. The more accurate pose estimate may then be used to identify matching relationships in successive images of the same pose but each containing a smaller field of view.

[0056] In some examples, structured information associated with the evidence of use may be used to transform or otherwise modify a 3D model of an object, for example, so that it fits a particular instance of the object. The modified 3D model may be used in various simulation and evaluation processes.

[0057] A variety of different approaches can be used to identify three-dimensional locations that correspond to two-dimensional locations in the image of the narrower FOV in order to modify the 3D model.

[0058] For example, in some implementations, process 300 may be performed multiple times using multiple images with diverse poses. For example, each performance of process 300 may use images of different narrower FOVs along with images of the same or different wider FOVs. The images of narrower FOVs may be taken from different poses but contain the same evidence of use in their respective fields of view. Comparison of the locations of the evidence of use identified in different performances of process 300 may be used to identify parallax errors inherent in those locations. If the different poses of the images are sufficiently diverse, three-dimensional structures of the evidence of use may be identified and used to deform or otherwise modify the 3D model.

[0059] As another example, three-dimensional depth data may be obtained from a model instance of an object and used with the image information to train a machine learning model to estimate three-dimensional structures associated with the traces of use. The three-dimensional depth data may be obtained using any of a variety of different coordinate measuring machines, including, for example, mechanical metrology probes, laser scanners, LIDAR sensors, etc. In general, the machine learning model may be trained using multiple images from a variety of poses and use positions identified from multiple performances of process 300 to estimate the three-dimensional depth data.

[0060] As another example, in some embodiments, the evidence of use may result from a well-constrained process. Examples include, for example, the repetitive wear of a gear or the repetitive loading of a shaft. The process constraints may be used to identify three-dimensional structures that correspond to the locations identified in process 300.

[0061] The subject matter embodiments and operations described herein may be implemented in digital electronic circuitry, including the structures disclosed herein and structural equivalents thereof, or in computer software, firmware, or hardware, or by any combination of one or more of them. The subject matter embodiments described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded in a computer storage medium for execution by or to control the operation of a data processing apparatus. Alternatively, or in addition, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to an appropriate receiver device for execution by the data processing apparatus. The computer storage medium may be or be included in a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Furthermore, while a computer storage medium is not a propagated signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be, or be contained within, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0062] The operations described herein may be realized as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or received from other sources.

[0063] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination or combinations of these. The apparatus may include special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). In addition to hardware, the apparatus may further include code that creates an execution environment for the computer program in question, such as code constituting a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform execution environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0064] A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may correspond to a file in a file system, but this is not required. A program may be stored in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in one file assigned to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subprograms, or code portions). A computer program may be deployed to be executed in one computer or in multiple computers, located at one site or distributed across multiple sites and interconnected by a communication network.

[0065] The processes and logic flows described herein may be implemented by one or more programmable processors executing one or more computer programs to perform operations by processing input data and generating output. The processes and logic flows may further be implemented by, and an apparatus may further be realized as, special purpose logic circuitry, such as, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0066] Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for performing operations according to the instructions, and one or more memory devices for storing instructions and data. Generally, a computer further includes one or more mass storage devices for storing data, such as, for example, a magnetic disk, a magneto-optical disk, or an optical disk, or is operatively coupled to receive data from, or transmit data to, the one or more mass storage devices, or both. However, a computer need not have such devices. Furthermore, a computer may be incorporated in another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0067] To provide for interaction with a user, embodiments of the subject matter described herein may be implemented in a computer that includes a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to a user, and a keyboard and pointing device, such as a mouse or trackball, by which the user may provide input to the computer. Other types of devices may be used to provide interaction with a user as well, for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form including acoustic, speech, or tactile input. Additionally, the computer may interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser at the user's client device in response to a request received from the web browser.

[0068] Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be realized in combination in one embodiment. Conversely, various features described in the context of one embodiment may also be realized in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may have been described herein above as operating in a particular combination, and may even initially be claimed as such, one or more features from the claimed combination may in some instances be excluded from the combination, and the claimed combination may relate to a subcombination or a variation of the subcombination.

[0069] Similarly, although operations are depicted in a particular order in the figures, this should not be understood to require that such operations be performed in the particular order shown, or in a sequential order, or that all of the shown operations be performed to achieve desirable results. In certain situations, multitasking and parallel processing may be beneficial. Furthermore, the separation of various system components in the above-described embodiments should not be understood to require such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0070] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some instances, the actions recited in the claims may be performed in a different order and still achieve desirable results. In addition, the steps depicted in the accompanying figures do not necessarily require the particular order or sequence shown to achieve desirable results. In certain implementations, multitasking and parallel processing may be beneficial.

Claims

1. 1. A method implemented by a data processing apparatus, the method comprising: projecting, based on a pose of an instance of the object in the relatively wide field of view image, a usage signature of the instance of the object in the relatively wide field of view image onto a 3D model of the object; Identifying 3D coordinates of the trace of use in the 3D model; estimating a relative pose of the instance of the object in the relatively narrow field of view image using the 3D coordinates of the trace of use in the 3D model based on a matching relationship between the trace of use in the relatively wide field of view image and the same trace of use in the relatively narrow field of view image; A method comprising:

2. computing a virtual location of the trace of use in the relatively narrow field of view image using the estimated pose and a projection of the trace of use onto the 3D model; comparing the virtual location with an actual location of the indication of use in the relatively narrow field of view image; Identifying a subset of the traces of use that are inappropriately projected onto the 3D model based on a comparison result; and The method of claim 1 further comprising:

3. deforming the 3D model using a second subset of the traces of usage appropriately projected onto the 3D model; The method of claim 2 further comprising:

4. and projecting improperly projected ones of the indications of use onto the 3D model of the object based on the relative poses of the instances of the object in the relatively narrow field of view images. The method of claim 2.

5. projecting the improperly projected traces of use onto the 3D model; identifying an area of ​​the relatively narrow field of view image that includes a first one of the improperly projected ones of the indications of use; matching the first of the improperly projected ones of the indicia of use with the first of the indicia of use in the relatively wide field of view image; projecting the first of the indications of use in the relatively wide field of view image onto the 3D model of the object; Including, The method according to claim 4.

6. the subset of improperly projected indicia of use is identified based on a misalignment between the virtual locations and the actual locations of the indicia of use in the relatively narrow field of view image. The method of claim 2.

7. The method further comprising: filtering the subset of the usage indications that are improperly projected onto the 3D model from the matching relationships to establish a proper subset of the matching relationships; re-estimating the relative pose of the instance of the object in the relatively narrow field of view image based on the subset of matching relations; The method of claim 2 , comprising:

8. projecting the traces of use in the relatively wide field of view images onto the 3D model includes identifying a pose in the relatively wide field of view images. The method of claim 1.

9. identifying a dominant color of the instance of the object in the relatively narrow field of view image; identifying areas in the relatively wide field of view image, in the relatively narrow field of view image, or in both the relatively wide field of view image and the relatively narrow field of view image that deviate from the dominant color; matching the identified areas to match the signs of use in the relatively wide field of view image with the signs of use in the relatively narrow field of view image; The method of claim 1 further comprising:

10. determining deviations from ideality in the instances of the object in the relatively narrow field of view image; matching the offset in the relatively narrow field of view image to the relatively wide field of view image to match the evidence of use in the relatively wide field of view image with the evidence of use in the relatively narrow field of view image; The method of claim 1 further comprising:

11. estimating a relative pose of the instance of the object in a plurality of relatively narrow field of view images; using the estimated pose to compute a virtual location of the same trace of use in the relatively narrow field of view image; The method of claim 1 , comprising:

12. 1. A method implemented by a data processing apparatus, the method comprising: annotating a 3D model of an object using usage evidence from two or more 2D images of an instance of the object; annotating the 3D model, receiving the 3D model and the 2D images, a first of the 2D images being a relatively wide field of view image of the instance and a second of the 2D images being a relatively narrow field of view image of the instance; matching visible signs of use in the relatively wide field of view image and the relatively narrow field of view image; projecting the indications of use in the relatively wide field of view image onto the 3D model of the object; estimating a pose of the instance in the relatively narrow field of view image using a projection of the trace of usage onto the 3D model and the matched trace of usage in the relatively wide field of view image and the relatively narrow field of view image; computing a virtual location of the trace of use in the relatively narrow field of view image using the estimated pose and a projection of the trace of use onto the 3D model; comparing the virtual locations with actual locations of the signs of use in the relatively narrow field of view image to identify improperly projected ones of the signs of use; removing the improperly projected ones of the indications of use from a projection onto the 3D model of the object; and A method comprising:

13. and projecting the improperly projected ones of the indications of use onto the 3D model of the object based on a relative pose of the instances of the object in images of a relatively narrow field of view. The method of claim 12.

14. projecting the improperly projected traces of use onto the 3D model; identifying an area of ​​the relatively narrow field of view image that includes a first one of the improperly projected ones of the indications of use; matching the first of the improperly projected ones of the indicia of use with the first of the indicia of use in the relatively wide field of view image; projecting the first of the indications of use in the relatively wide field of view image onto the 3D model of the object; Including, The method of claim 13.

15. the improperly projected subset of the indications of use is identified based on a misalignment between the virtual locations and the actual locations of the indications of use in the relatively narrow field of view image. The method of claim 12.

16. Matching the traces of use, identifying a dominant color of the instance of the object in the relatively narrow field of view image; identifying areas in the relatively wide field of view image, in the relatively narrow field of view image, or in both the relatively wide field of view image and the relatively narrow field of view image that deviate from the dominant color; Including, The method of claim 12.

17. Matching the traces of use, determining deviations from ideality in the instances of the object in the relatively narrow field of view image; matching the displacement in the relatively narrow field of view image to the relatively wide field of view image; Including, The method of claim 12.

18. and transforming the 3D model using a suitable projected one of the traces of use from a projection of the object onto the 3D model. The method of claim 12.

19. At least one computer-readable storage medium encoded with executable instructions that, when executed by at least one processor, cause the at least one processor to perform operations for annotating a 3D model of an object using usage evidence from two or more 2D images of an instance of the object, the annotating the 3D model comprising: receiving the 3D model and the 2D images, a first of the 2D images being a relatively wide field of view image of the instance and a second of the 2D images being a relatively narrow field of view image of the instance; matching visible signs of use in the relatively wide field of view image and the relatively narrow field of view image; projecting the indications of use in the relatively wide field of view image onto the 3D model of the object; estimating a pose of the instance in the relatively narrow field of view image using a projection of the trace of usage onto the 3D model and the matched trace of usage in the relatively wide field of view image and the relatively narrow field of view image; computing a virtual location of the trace of use in the relatively narrow field of view image using the estimated pose and a projection of the trace of use onto the 3D model; comparing the virtual locations with actual locations of the signs of use in the relatively narrow field of view image to identify improperly projected ones of the signs of use; removing the improperly projected ones of the indications of use from a projection onto the 3D model of the object; and Including, A computer-readable storage medium.