Method for selecting surface points from a CAD model to position an industrial 3D object, the method being applied to the positioning of an industrial 3D object and to an augmented reality system using the 3D object thus positioned

Abstract

A method for automatically selecting 3D points from a CAD model for 3D positioning of an industrial object comprises the steps of: generating a geometric model from the CAD model; pre-specifying a reference view from a priori knowledge of the current view point of a camera; acquiring a plurality of images taken from a view of interest of the object by the camera; associating each acquired image with the reference view so as to form a subset (partition) of images and identifying a reference image for each partition; and selecting n 3D surface points for each partition. In particular, the application of positioning industrial 3D objects in augmented reality devices.

Description

Technical Field

[0001] This invention relates to a method for selecting surface points from a CAD model to locate industrial 3D objects. It also aims to apply this method to the localization of industrial 3D objects, and to augmented reality systems utilizing the 3D objects located thereby.

[0002] The field of this invention is particularly the field of augmented reality solutions, especially for assisting in the installation and assembly of industrial components. Background Technology

[0003] Computer viewing and machine learning techniques already exist for images acquired by RGB monocular viewing systems. These techniques are both inexpensive and easy to implement because they do not rely on the use of multi-sensor or 3D sensors. Furthermore, they do not require instrumentation of the scene of interest or the equipping of it with sensors, beacons, or markers.

[0004] However, when the camera device used is fully mobile, difficulties arise in the presence of highly reflective and poorly textured objects, as well as under highly variable lighting conditions. This is especially true in industrial environments, where millimeter-level positioning accuracy is sought due to the potentially highly variable background of the scene of interest.

[0005] The goal is for learning-based 6D localization systems to have the ability to estimate the pose of a camera device relative to a known object, regardless of the background, position, and orientation of the object and camera device under all lighting conditions.

[0006] One existing approach to meet this expectation implements the concept of BOF (“feature bag”) representation, which involves the step of extracting local features.

[0007] The first phase, known as the learning phase, is performed offline to build a representation model. The second phase, known as the testing phase, is performed online and allows for 3D localization of the objects.

[0008] During the first phase, local descriptors were extracted from the training image set to construct a “visual vocabulary bag” commonly referred to as an acronym BOF (feature bag). These descriptors are extracted from regions of interest located within image areas containing objects. Local descriptors are essentially 2D observations of 3D points reconstructed through triangulation or by backprojection of 2D points onto the surface of a CAD model.

[0009] During the second phase, feature extraction and matching with the "visual bag", object recognition, and 6D pose estimation via 2D / 3D matching are performed consecutively.

[0010] However, this method, which allows for object recognition and also estimates the camera's pose, has the drawback of only working well when the background changes very little. It is not robust to large changes in object appearance. It may even fail if the object has poor texture or is highly reflective, because not only will the region of interest change, but the local descriptor will also change.

[0011] Over the past decade, a confluence of computational power and key theoretical advances has enabled the emergence of deep learning techniques based on CNN (Convolutional Neural Network) models. This second CNN-based approach includes a first convolutional portion that serves as an automatic extractor based on the most relevant features from the training data.

[0012] The second part of the CNN involves optimizing the network coefficients to minimize the output classification error. Simultaneously, both parts of the CNN are trained as follows: learning the coefficients of the convolutional kernels to extract relevant features, and learning the correct combination of these features.

[0013] Two main categories of CNN-based methods should be distinguished. The first category treats the 6D pose estimation problem as a pose classification or regression problem. These methods do not provide very accurate pose estimates. The second category of methods is based on intermediate representations. To achieve higher accuracy, these methods rely on the projection of 3D points in the localization image. The pose can be easily calculated by correlating 2D and 3D data based on the 3D coordinates of these points.

[0014] The paper "Estimating 6D Pose From Localizing DesignatedSurface Keypoints" by Zelin Zhao et al. (arXiv: 1812.01387vl [cs.CV] December 4, 2018) discloses a technique for estimating 6D pose from RGB images. In this technique, a set of surface points corresponding to regions of interest detected in the image are designated as keypoints, a keypoint detector (KPD) is trained to locate these keypoints, and then a PnP-type algorithm for calculating 6D pose based on 2D data is executed.

[0015] The paper by Zelin Zhao et al. did not address the problem of selecting the most relevant surface points. The steps involved in selecting these points are crucial for overcoming the limitations of use under highly variable background and lighting conditions, which is often the case in industrial environments.

[0016] The object of this invention is to provide a method for selecting surface points that can be used in industrial environments, which enables exhaustive learning with generalization capabilities beyond the learning data. Summary of the Invention

[0017] This objective is achieved through a method for automatically selecting 3D points from a CAD model for 3D positioning of industrial objects, the method comprising the following steps:

[0018] - Generate a geometric model from a CAD model, including sampling a set of 3D points extracted from the straight edges of the CAD model to generate 3D surface points.

[0019] -A reference viewpoint is pre-specified based on prior knowledge of the camera device's current viewpoint.

[0020] - Multiple images of the object are acquired from a viewpoint of interest using a camera device. Each acquired image is associated with a reference viewpoint to form a subset (partition) of images, and a reference image for each partition is identified.

[0021] - Select n 3D surface points for each partition, including the following steps:

[0022] - The camera device pose is refined using a geometric model, and the association between 3D surface points and corresponding 2D contour points of the refined pose is identified.

[0023] - A score for each 3D surface point is calculated based on the number of times it matches a contour point. 3D surface points in each reference image are then projected and grouped.

[0024] - For each selection in the partition, select the n 3D surface points with the best scores.

[0025] To perform 3D positioning of an object captured by a camera device, six degrees of freedom of the camera device's pose relative to the object are measured.

[0026] When the selection method according to the invention is implemented in localization based on six degrees of freedom of a convolutional neural network, it may also advantageously include the following steps:

[0027] - Train the first convolutional neural network to identify the viewpoint of interest of an object among a set of reference viewpoints;

[0028] - For each partition, learn the heatmap corresponding to the 2D projection of the selected 3D surface point.

[0029] The selection method according to the invention may further include the following steps: identifying the closest viewpoint among six predefined viewpoints automatically calculated based on the 3D bounding box of the 3D object and camera device calibration parameters using a first convolutional neural network.

[0030] The selection method may also include a partition sequence, which includes the following steps:

[0031] - Annotate the images for each learning sequence by initiating 3D tracking.

[0032] - 2D projection and / or visibility tests based on 3D surface points associate each image with the closest reference viewpoint.

[0033] - Identify p reference images obtained from the viewpoints closest to the p reference viewpoints, thereby generating p partitions of the acquired images.

[0034] The selection method according to the invention may further include the steps of: associating partitions with each image for the purpose of teaching multiple types of detectors and for learning sequences, during which:

[0035] - Teaching multiple types of detectors allows for the detection of industrial objects and prediction of their distribution.

[0036] - For each distribution, the keypoint detector is trained based on heatmap localization.

[0037] It is important to note that all steps of the selection method according to the invention can be implemented by one or more computers performing one or more software applications that implement these steps.

[0038] According to another aspect of the invention, an application of the selection method according to the invention for locating industrial 3D objects, the geometric model of which is generated offline, is proposed. This application includes the following steps:

[0039] - Use a geometric model to verify and refine the pose; this step requires prior initialization.

[0040] - Generate the 3D location of the object.

[0041] - Construct an appearance model of the object.

[0042] In another application of the selection method according to the present invention, the other application may further include the following steps:

[0043] - Construct an appearance model based on this refined pose.

[0044] - Extract features from subsequent frames and match them with the appearance model.

[0045] - Estimate the pose of 3D objects

[0046] - Determine the confidence score of the attitude estimate.

[0047] The attitude estimation step can be followed by the following steps, which are constrained by the confidence score level:

[0048] - For confidence scores below a first predetermined threshold, perform 3D relocalization based on the already generated set of appearance models.

[0049] - For the confidence score between the first threshold and the predetermined value, refine it using a geometric model, and then verify the pose.

[0050] - For confidence scores higher than the second predetermined value, confirm the 3D location.

[0051] The success or failure of the 3D relocalization step can further lead to the new construction of the appearance model and the 3D localization of the object, or to the reinitialization of the pose verification and refinement steps.

[0052] The initialization or re-initialization step before performing pose verification and refinement steps can also implement an initialization module based on a convolutional neural network architecture.

[0053] According to another aspect of the invention, an augmented reality system is proposed that utilizes a 3D object located by implementing a method for selecting surface points according to the invention.

[0054] The resulting industrial object positioning tools can optimize the overall performance of processes with high human complexity by providing digital assistance to guide field operations and capture real-world conditions. This enables the optimization of the execution, monitoring, and engineering of production, control, and maintenance processes.

[0055] This digitally assisted operational tool enables the handling of industrial 3D objects from the engineering design office to the shop floor and vice versa. In the shop floor, the tool assists field operators in performing complex tasks and collects field data and control results for later use in the design office for optimization purposes. Digital data, such as 3D models, work instructions, and contextual information, is then transmitted through the tool.

[0056] definition

[0057] CNN:

[0058] CNN stands for Convolutional Neural Network.

[0059] Edge (3D surface point):

[0060] An edge is a 3D point extracted from the straight edge of a 3D model. Each edge has a 3D orientation of the straight edge from which it was extracted. An edge is a point that can be represented by a contour point in a 2D image of a real object.

[0061] Heat map:

[0062] A heatmap is a graphical representation of statistical data that maps the intensity of a variable onto a series of hues or colors on a two-dimensional matrix. CNN prediction techniques for heatmaps capture both local and global features to construct accurate KPDs.

[0063] Keypoint Detector (KPD):

[0064] The software module allows for the localization of 2D points by predicting their heatmaps.

[0065] Matching contour points:

[0066] 3D surface points (edges) projected from the camera device's pose and corresponding to contour points in the image are here considered to be matching contour points.

[0067] Appearance Model:

[0068] An appearance model is an image representation characterized by a set of local appearance descriptors.

[0069] Geometric model:

[0070] The geometric model is generated directly from the object's CAD drawing. Its characteristic is a set of 3D surface points oriented in 3D.

[0071] PnP:

[0072] PnP stands for "Perspective n Points" and refers to an algorithm for estimating the pose of a calibrated camera device from a set of n 3D points in the world and their corresponding 2D projections in an image. Attached Figure Description

[0073] [ Figure 1 ] Figure 1 This is a block diagram illustrating an embodiment of the 3D positioning method according to the present invention.

[0074] [ Figure 2 ] Figure 2 This is a block diagram of a 3D tracking operation that implements methods for selecting surface points and 3D positioning;

[0075] [ Figure 3 ] Figure 3 A first embodiment of the surface point selection method for photographing attitude according to the present invention, applied to industrial aerospace components, is shown;

[0076] [ Figure 4 ] Figure 4 A second example of a surface point selection method for photographing poses according to the present invention, applied to a car hood, is shown;

[0077] [ Figure 5 ] Figure 5 A third embodiment of the surface point selection method for photographing posture according to the present invention, applied to automotive components, is shown;

[0078] [ Figure 6 ] Figure 6A set of images of the same automotive component processed in the surface point selection method according to the invention are shown, taken under variable lighting, background and attitude conditions;

[0079] [ Figure 7 ] Figure 7 This illustrates a set of edgelets automatically selected in the form of surface points on the straight edges of an object after the clustering step in the surface point selection method according to the invention; and

[0080] [ Figure 8 ] Figure 8 An example use of an augmented reality device is shown to implement 3D positioning, which implements a surface point selection method according to the present invention. Detailed Implementation

[0081] Reference Figure 1 A geometric model is generated based on CAD. This model is used to refine and then verify the pose of the camera device used to capture real objects. Starting from this step, an appearance model is built, and 3D positioning is performed.

[0082] As a non-limiting example, the refinement may include optimizing the camera device's pose to maximize the number of matches between 2D and 3D points. Once the match ratio exceeds a predetermined threshold, such as 50% of the 3D points visible from the camera device's angle, the camera device's pose is verified.

[0083] For subsequent images, appearance features are extracted, and then feature matching is performed based on the final appearance model. This matching is followed by pose estimation.

[0084] If the attitude estimation results in a high confidence score, the 3D localization step is performed again.

[0085] As a non-limiting example, the confidence score can be determined as the score used to match 2D features between the final appearance model and the image being processed. This score can be calculated as the ratio of the number of matched primitives to the total number of detected primitives.

[0086] If the pose estimation results in an average confidence score, then a refinement and then pose validation step is performed, and a new appearance model is generated.

[0087] If pose estimation results in a low confidence score, then 3D relocalization is performed based on all appearance models.

[0088] If the 3D relocalization fails, a CNN-based module is used to estimate the pose, followed by pose refinement and verification steps.

[0089] If the relocation is successful, rebuild the appearance model again. (Refer to...) Figure 2The CNN inference method implemented in the point selection method according to the invention includes detecting partitions of objects in an image, followed by a step of loading a specific network for heatmap localization. The location of the heatmap is predicted, followed by PnP pose estimation. This is followed by a step of refining and validating the pose using a geometric model, which leads to 3D tracking.

[0090] exist Figure 3 In the actual embodiment shown, the industrial 3D object positioning system according to the present invention is implemented in workstation 10 to position industrial component 1.

[0091] In "offline" mode, operator 3 has placed one or more mobile camera devices 2 to capture a set of views of component 1. A 3D representation of component 1 from the CAD model appears on the screen 11 of workstation 10, and on this representation, there is a set of surface points 20 that have been selected on the straight edges of component 1. These selected surface points will later be used to locate the 3D component in an augmented reality device worn by an operator on site, whose task is to intervene in an industrial component of the type that has been processed by the selection method according to the invention.

[0092] The device for photographing and selecting surface points according to the method of the present invention can be implemented to process all kinds of industrial objects, such as automobile engine hoods. Figure 4 or industrial parts obtained by molding or injection. Figure 5 ).

[0093] The image partitioning technique implemented in the selection method according to the present invention will now be described. These images may have been taken under different lighting, background, or pose conditions, such as... Figure 6 As shown.

[0094] This partitioning technology includes the following steps:

[0095] - Annotate the images in each learned video sequence by initiating 3D tracking and performing image pose estimation.

[0096] - Based on visibility testing and edge-based 2D projection, the nearest reference viewpoint is associated with each image.

[0097] - Identify the p reference images associated with the p nearest reference viewpoints in the same way.

[0098] - Based on visibility testing and 2D projection of edges, the most similar reference image is associated with each image, such as... Figure 5 As shown.

[0099] To select surface points, the following operations are performed for each partition:

[0100] - A score for each edge is calculated based on the number of times the refined camera pose from each image appears as an "interior point" (matching edge-contour).

[0101] - Project and cluster edges in the p-reference image that have scores above a threshold to ensure good spatial distribution and retain only n edge partitions.

[0102] - For each choice in the p-distribution, there are n margins with the best score; such as Figure 7 As shown.

[0103] Therefore, we obtain n edges selected for each partition.

[0104] As a non-limiting example, the score can simply be calculated as the number of occurrences. If sequences taken in different environments (light or location) are available, the sum of occurrences in each sequence can be multiplied.

[0105] A description of an implementation of the learning method of the convolutional neural network according to the present invention will now be given.

[0106] Training data preparation

[0107] First, partitions are associated with each image in order to teach multi-class detectors, during this step:

[0108] - Assign a class to each image

[0109] - Object face detection is considered a multi-class detection problem.

[0110] For each partition, project the n selected edges and generate a heatmap image.

[0111] study

[0112] The system teaches multiple types of detectors, enabling the detection of industrial objects and prediction of their distribution. For each distribution, the KPD is taught based on heatmap localization.

[0113] The selection method according to the present invention can be applied to the fields of digital assistance, execution of virtual production instruction sheets, assembly traceability, information declaration, and consistency checks, and especially as... Figure 8 The augmented reality field is shown. Due to the relevant selection of surface points in complex industrial systems such as aircraft engines, precise positioning of 3D objects is possible, providing maintenance operators who must intervene in these complex objects with greatly enhanced user comfort and highly accurate displayed information.

[0114] Of course, the present invention is not limited to the embodiments described above, and many other embodiments can be conceived without departing from the scope of the present invention.

Claims

1. A method for automatically selecting 3D points from a CAD model for 3D positioning of an industrial object, the method comprising the following steps: Generating a geometric model from the CAD model involves sampling a set of 3D points extracted from the straight edges of the CAD model to generate 3D surface points. A reference viewpoint is pre-designated based on prior knowledge of the current viewpoint of the camera device. The camera device acquires multiple images of the object from viewpoints of interest. Each of the plurality of images acquired from the viewpoint of interest is associated with one of the reference viewpoints to form a plurality of image subsets, i.e., each corresponding to a partition of one of the reference viewpoints, and a reference image representing each partition is identified. For each partition, n 3D surface points are selected, which includes the following steps: The camera device pose is refined using the geometric model by maximizing the number of matches between 2D contour points and 3D surface points from the image in the partition, and the association between 3D surface points and 2D contour points corresponding to the refined pose is identified. The score for each 3D surface point is calculated based on the number of times the 3D surface point matches the 2D contour point. 3D surface points are projected onto a reference image representing the partition and clustered. Select the n 3D surface points with the best scores from the clustered 3D surface points.

2. The method according to claim 1, wherein the method is implemented in six-degree-of-freedom localization based on a convolutional neural network, further includes the following steps: The first convolutional neural network is trained to identify the viewpoint of interest for the object among a set of reference viewpoints; For each partition, learn the heatmap corresponding to the 2D projection of the selected 3D surface point.

3. The method according to claim 2, characterized in that, The method further includes the following step: using the first convolutional neural network to identify the viewpoint closest to the current viewpoint of the camera device among six predefined viewpoints automatically calculated based on the six degrees of freedom positioning, according to the 3D bounding box of the 3D object and the calibration parameters of the camera device.

4. The method according to claim 3, characterized in that, The method further includes image partitioning, which comprises the following steps: Image annotation is performed for each learning video sequence by initiating 3D tracking. Based on 2D projection and / or visibility tests of 3D surface points, the nearest reference viewpoint is associated with each image. Identify the p reference images acquired from the closest viewpoint among the p reference viewpoints, thereby generating p partitions of the acquired image.

5. The method according to claim 4, characterized in that, The method further includes the step of associating partitions with each image for use in teaching multi-class detectors.

6. The method according to any one of claims 2 to 5, characterized in that, The method further includes a learning sequence, during which: Multiple types of detectors are trained to allow the detection of the industrial objects and the prediction of their distribution. For each distribution, the keypoint detector is trained based on the location of the heatmap.

7. A method for locating an industrial 3D object, the method comprising performing the method according to any one of claims 1 to 6, wherein, The geometric model of the industrial 3D object is generated offline, and the method for locating the industrial 3D object further includes the following steps: The geometric model is used to verify and refine the pose; this step is pre-initialized. Generate the 3D positioning of the object. Construct the appearance model of the object.

8. The positioning method according to claim 7, characterized in that, The method also includes the following steps performed online: The appearance model is constructed based on such a detailed posture. Features are extracted from subsequent frames and matched with the appearance model. Estimate the pose of the 3D object. Determine the confidence score of the attitude estimate.

9. The positioning method according to claim 8, characterized in that, The attitude estimation step is followed by a step constrained by the level of the confidence score in the following manner: For confidence scores below a first predetermined threshold, 3D relocalization is performed based on the already generated set of appearance models. For the confidence score between the first predetermined threshold and a predetermined value, the geometric model is used to refine and then verify the pose. The 3D positioning is confirmed if the confidence score is greater than the second predetermined value.

10. The positioning method according to claim 9, characterized in that, The success or failure of the 3D relocalization step leads to the reconstruction of the appearance model and the 3D localization of the object, or to the reinitialization of the pose verification and refinement steps.

11. The positioning method according to claim 10, characterized in that, The initialization or re-initialization step prior to performing the pose verification and refinement steps implements an initialization module based on the convolutional neural network architecture.

12. An augmented reality system utilizing a 3D object located by implementing the method according to any one of claims 1 to 6, the augmented reality system comprising: At least one processor; as well as Memory connected to the at least one processor, The memory stores instructions executable by the at least one processor, and when executed by the at least one processor, the instructions cause the at least one processor to implement the method according to any one of claims 1 to 6.

Images