Method for selecting surface points from a CAD model for locating industrial 3D objects, application of this method for locating industrial 3D objects, and augmented reality system using the 3D objects thus located

Description

FIELD OF INVENTION

[0001] The present invention relates to a method for selecting surface points from a CAD model for the localization of industrial 3D objects. It also aims at the application of this method to the localization of industrial 3D objects as well as an augmented reality system using 3D objects thus localized.

[0002] The field of the invention is in particular that of augmented reality solutions, in particular for assistance in the assembly and assembly of industrial parts. STATE OF THE ART

[0003] Computer vision and machine learning techniques already exist that can be applied to images acquired by a monocular RGB vision system. These techniques are both inexpensive and easy to implement because they do not rely on multi-sensor or 3D sensors. Furthermore, they do not require instrumenting the scene of interest and equipping it with sensors, beacons, or markers.

[0004] But implementation difficulties arise when the camera used is completely mobile, in the presence of highly reflective and poorly textured objects and in highly variable lighting conditions. This is particularly the case in industrial environments where the backgrounds of the scenes of interest can vary greatly, while millimeter-level localization precision is sought.

[0005] A learning-based 6D localization system is expected to have the ability to estimate the camera pose relative to a known object regardless of the background, position and orientation of the object and camera, and regardless of the lighting conditions.

[0006] An existing approach to meet this expectation implements the concept of BOF (Bag of Features) representation which involves a step of extracting local features.

[0007] A first phase, called learning, is carried out offline to establish a representation model. A second phase, called testing phase, is carried out online and allows the 3D localization of the object.

[0008] In the first phase, local descriptors are extracted from the training image set to construct a “bag of visual words,” commonly referred to as BOF (Bag Of Features). These descriptors are extracted from areas of interest located in the image region where the object is present. Local descriptors are in fact 2D observations of 3D points that are reconstructed by triangulation or by backprojecting 2D points onto the surface of the CAD model.

[0009] During the second phase, the steps of feature extraction and matching with the "bag of visual words", object recognition, and 6D pose estimation by 2D / 3D matching are successively carried out.

[0010] This approach, which allows object recognition and also camera pose estimation, has the disadvantage of only working well if the context changes little. It is not robust in the event of a large change in the object's appearance. It may even fail if the object is poorly textured or highly reflective because not only do the areas of interest change, but also the local descriptors.

[0011] Over the past decade, the combination of computational capabilities and key theoretical advances has led to the emergence of techniques based on deep learning using CNN (Convolutional Neural Network) models. This second CNN-based approach includes a first convolutional part functioning as an automatic extractor of the most relevant features based on the training data.

[0012] The second part of the CNN consists of optimizing the network coefficients to minimize the output classification error. Both parts of the CNN are trained simultaneously: we learn both the coefficients of the convolution kernels to extract relevant features, and the right combination of these features.

[0013] There are two main categories of CNN-based methods. Methods that treat the 6D pose estimation problem as a classification or pose regression problem. These methods are not capable of very accurate pose estimation. Methods in the second category rely on an intermediate representation. To achieve greater accuracy, these methods rely on the location of the projection of 3D points in the images. From the 3D coordinates of these points, a pose can be easily calculated by associating 2D-3D data.

[0014] The paper "Estimating 6D Pose From Localizing Designated Surface Keypoints" by Zelin Zhao et al. (arXiv:1812.01387v1 [cs.CV] 4 Dec 2018) discloses a technique for estimating 6D pose from an RGB image, in which a set of surface points that correspond to areas of interest detected in the images are designated as keypoints, a keypoint detector (KPD) is trained to locate them, and then a PnP algorithm is performed to calculate the 6D pose based on the 2D data. The paper by Zelin Zhao et al. does not address the problem of selecting the most relevant surface points. The step of selecting these points is a crucial step to overcome the limitations of use in highly variable background and lighting conditions, which is often the case in industrial environments.

[0015] The paper Loesch Angelique ET AL, "Localization of 3D objects using model-constrained SLAM", MACHINE VISION AND APPLICATIONS, vol. 29, n. 7, June 30, 2018, pages 1041-1068, describes a method for localizing 3D objects using a model and oriented 3D contour points called "edgelets".

[0016] The aim of the present invention is to propose a method for selecting surface points which can be used in an industrial environment and which makes it possible to carry out exhaustive learning with a power of generalization beyond the learning data. SUMMARY OF THE INVENTION

[0017] This objective is achieved with a process of automatic selection of 3D points from a CAD model, for the 3D localization of an industrial object, comprising the following steps: generation of a geometric model from the CAD model, comprising sampling a set of 3D points extracted from sharp edges of said CAD model, so as to generate 3D surface points, designation in advance of reference viewpoints from a priori knowledge of the current viewpoints of the camera, acquisition by means of a camera of a plurality of images taken from viewpoints of interest of said object, association of each acquired image with a reference viewpoint to form subsets of images (partitions) and identification for each partition of a reference image, selection for each partition of the n 3D surface points, comprising the following steps: refinement of the camera poses using the geometric model and identification of the associations of 3D surface points with 2D contour points which correspond to the refined poses,calculating a score for each 3D surface point based on the number of times this 3D surface point is a matching contour point, projecting and grouping the 3D surface points in each reference image, selecting the n 3D surface points with the best score for each of the partitions.

[0018] To perform a 3D localization of an object filmed by a camera, we measure the six degrees of freedom of the camera's pose relative to this object.

[0019] When the selection method according to the invention is implemented in a localization according to the six degrees of freedom based on a convolutional neural network, it can further advantageously comprise the steps of: learning a first convolutional neural network for recognizing a viewpoint of interest of the object among a set of reference viewpoints; for each partition, learning heat maps corresponding to the 2D projections of the selected 3D surface points.

[0020] The selection method according to the invention may further comprise a step for identifying by the first convolutional neural network the closest viewpoint among six predefined viewpoints calculated automatically as a function of a bounding box (3D) of the 3D object and camera calibration parameters.

[0021] This selection method may further comprise a partitioning sequence comprising the following steps: annotation of images for each training sequence by launching a 3D tracking, association with each image of the closest reference viewpoint based on 2D projections of the 3D surface points and / or visibility tests, identification of p reference images acquired from the viewpoint closest to p reference viewpoints, thus carrying out p partitions of the acquired images.

[0022] The selection method according to the invention may further comprise a step of associating a partition with each image with the aim of learning a multi-class detector, as well as a learning sequence during which: we learn a multi-class detector to detect the industrial object and predict its distribution, for each distribution, we train a key point detector based on the location of heat maps.

[0023] It is important to note that all the steps of the selection method according to the invention can be implemented by one or more computers running one or more software programs implementing these steps.

[0024] According to another aspect of the invention, an application of the selection method according to the invention is proposed for the localization of an industrial 3D object of which a geometric model is generated offline, comprising the following steps: validation and refinement of a pose using the geometric model, this step being subject to prior initialization, generation of a 3D location of said object, construction of an appearance model of said object.

[0025] In another application of the selection method according to the invention, the latter may further comprise the following steps carried out online: constructing an appearance model from the pose thus refined, extracting features in subsequent frames and matching with said appearance model, estimating a pose for the 3D object, determining a confidence score for said pose estimation.

[0026] The pose estimation step can be followed by a step that is conditioned by the confidence score level as follows: for a confidence score lower than a first predetermined threshold value, performing a 3D relocation based on a set of appearance models already produced, for a confidence score between the first threshold value and a predetermined value, refinement then validation of the pose using the geometric model, for a confidence score higher than the second predetermined value, confirmation of the 3D location.

[0027] The success or failure of the 3D relocation step can further lead respectively either to a new construction of an appearance model and a 3D localization of the object, or to a reset of the validation and pose refinement step.

[0028] The initialization or reinitialization step prior to the execution of the pose validation and refinement step may further implement an initialization module based on a convolutional neural network architecture.

[0029] According to yet another aspect of the invention, an augmented reality system is proposed using 3D objects located by implementing the surface point selection method according to the invention.

[0030] The resulting industrial object location tool can be implemented to optimize the overall performance of highly complex processes for humans, by providing digital assistance to guide field operations and capture the reality on the ground. This optimizes the execution, monitoring and engineering of production, control and maintenance processes.

[0031] This digitally assisted operating tool allows for the processing of industrial 3D objects from an engineering design office to a workshop and vice versa. In this workshop, the tool helps an operator in the field with complex tasks and collects field data and control results, which are then used in the design office for optimization purposes. Digital data, such as 3D models, work instructions, and contextualized information, are then transferred via the tool. DEFINITIONS CNN:

[0032] CNN (Convolutional Neural Network) is a convolutional neural network. Edgelets (3D surface points) :

[0033] Edgelets are 3D points extracted from the sharp edges of the 3D model. Each edgelet has the 3D orientation of the sharp edge from which it was extracted. Edgelets are points that can provide contour points in 2D images of the real object. Heatmap:

[0034] A Heatmap (or heat map) is a graphical representation of statistical data that maps the intensity of a variable quantity to a range of tones or a color swatch on a two-dimensional matrix. The CNN Heatmap prediction technique captures local and global features to construct accurate KPDs. Key Point Detector (KPD):

[0035] Software module allowing the localization of 2D points by predicting their heat maps. Correspondence contour point:

[0036] Correspondence edge points are considered here to be 3D surface points (edgelets) that are projected from the camera pose and correspond to edge points in the images. Appearance model:

[0037] An appearance model is an image representation characterized by a set of local appearance descriptors. Geometric model:

[0038] A geometric model is generated directly from the CAD of the object. It is characterized by a set of 3D surface points oriented in 3D. PnP:

[0039] PnP stands for Perspective-n-Point and concerns algorithms for estimating the pose of a calibrated camera from a set of n 3D points in the world and their corresponding 2D projections in the image. BRIEF DESCRIPTION OF THE FIGURES

[0040] [FIG.1]The figure 1 is a functional diagram of an exemplary embodiment of the 3D localization method according to the invention; [FIG.2]The figure 2 is a functional diagram of a 3D tracking operation implementing the methods of surface point selection and 3D localization; [FIG.3]The figure 3 illustrates a first example of implementation of the method for selecting surface points according to the invention for taking poses, applied to an industrial aeronautical part; [FIG.4]The figure 4illustrates a second example of sequences for implementing the method for selecting surface points according to the invention for taking poses, applied to an automobile casing; [FIG.5]The Figure 5 represents a third example of implementation of the method of selecting surface points according to the invention for taking poses, applied to an automobile part; [FIG.6]The figure 6 illustrates a set of images of the same automotive part processed in the surface point selection method according to the invention, which were taken under variable lighting, background and pose conditions; [FIG.7]The figure 7 illustrates a set of edgelets automatically selected as surface points on the sharp edges of the object, after the clustering step in the surface point selection method according to the invention; and [FIG.8]the figure 8illustrates an example of use of augmented reality equipment implementing 3D localization implementing the surface point selection method according to the invention. DETAILED DESCRIPTION

[0041] In reference to the figure 1 , from the CAD, a geometric model is generated. A refinement and then a validation of the pose of the camera which films the real object are carried out using this model. From this step, a construction of an appearance model and a 3D localization are carried out.

[0042] As a non-limiting example, the refinement may consist of optimizing the camera pose in order to maximize the number of correspondences between 2D points and 3D points. The camera pose is validated when there is a ratio of correspondences greater than a predetermined threshold, for example 50% of the 3D points visible from the camera's point of view.

[0043] For subsequent images, appearance features are extracted and then feature matching is performed based on the last appearance model that was built. This matching is followed by pose estimation.

[0044] If the pose estimation leads to a high confidence score, the 3D localization step is performed again.

[0045] As a non-limiting example, the confidence score can be determined as a 2D feature matching score between the last appearance model and the image being processed. This score can be calculated as the ratio of the number of matched primitives to the total number of detected primitives.

[0046] If the pose estimation leads to an average confidence score, we perform the refinement step then validation of the pose and we generate a new appearance model.

[0047] If the pose estimation leads to a low confidence score, a 3D relocation is performed based on all the appearance models.

[0048] If this 3D relocation fails, we use the CNN-based module for pose estimation, then we perform the pose refinement and validation step.

[0049] If the relocation is successful, we again build an appearance model. With reference to the figure 2 , the CNN inference method implemented in the point selection method according to the invention comprises a detection of the partition of an object in an image, followed by a step of loading the specific localization network of Heatmap. A prediction of the position of the heatmaps is carried out, followed by a pose estimation by PnP. This is followed by a step of refinement and validation of the pose by the geometric model which leads to 3D tracking.

[0050] In a practical example of implementation illustrated by the figure 3 , an industrial 3D object localization system according to the invention is implemented in a workstation 10 in order to locate an industrial part 1.

[0051] In “offline” mode, an operator 3 has arranged one or more mobile cameras 2 in order to take a set of views of the part 1. A 3D representation of the part 1 from the CAD model appears on the screen 11 of the workstation 10, and on this representation a set of surface points 20 which have been selected on sharp edges of the part 1. It is these selected surface points which will be used subsequently for the location of the 3D part in augmented reality equipment worn by an operator in the field whose mission is to intervene on an industrial part of the type that has been processed by the selection method according to the invention.

[0052] This equipment for taking pictures and selecting surface points implementing the method according to the invention can be used for the treatment of all kinds of industrial objects, for example an automobile engine casing ( figure 4 ) or an industrial part obtained by molding or injection ( Figure 5 ).

[0053] We will now describe an image partitioning technique implemented in the selection method according to the invention. These images may have been taken under variable lighting, background or pose conditions, as illustrated in figure 6 The partitioning technique includes the following steps: annotation of images for each training video sequence by launching a 3D tracking, performing image poses, association with each image the closest reference viewpoint based on visibility tests and 2D projections of the edgelets, identification, in the same way, of p reference images associated with the p closest reference viewpoints, association with each image the most similar reference image based on visibility tests and 2D projections of the edgelets, as illustrated in Figure 5 .

[0054] For the selection of surface points, we carry out for each partition: a calculation of a score for each edgelet based on the number of occurrences as being " inline» (edgelet-contour matching) from a refined camera pose for each image, a projection and clustering of edgelets having a score above the threshold in p reference images to ensure a good spatial distribution and keep only n edgelet partitions, a selection of the n edgelets having the best score for each of the p distributions; as illustrated in figure 7 .

[0055] We thus obtain n edgelets selected for each partition.

[0056] As a non-limiting example, this score can be simply calculated as the number of occurrences. If we have sequences taken in different environments

[0057] (light or positioning), we can perform the product of the sum of the occurrences in each of the sequences.

[0058] We will now describe an example of implementing training of a convolutional neural network implemented in the selection method according to the invention. Preparing training data

[0059] We first associate a partition with each image in order to learn a multi-class detector, a step during which: we assign a class to each image we treat the object face detection problem as a multi-class detection problem

[0060] For each partition, we project the n selected edgelets and generate images of heatmaps. Learning

[0061] We learn a multi-class detector to detect the industrial object and predict its distribution. For each distribution, learn a KPD based on the location of heatmaps.

[0062] The selection method according to the invention can be applied in the field of digital assistance, execution of virtual production instruction sheets, assembly traceability, information surveys and conformity checks, and particularly in the field of augmented reality as illustrated by figure 8 . Thanks to a relevant selection of surface points of a complex industrial system such as an aircraft reactor, the precise localization of 3D objects is made possible, which provides great comfort of use and high precision in the information displayed for a maintenance operator having to intervene on these complex objects. Of course, the present invention is not limited to the embodiment which has just been described and many other embodiments can be envisaged without departing from the scope of the invention as defined in the claims.

Claims

1. A computer-implemented method for automatically selecting 3D points from a CAD model, for 3D locating an industrial object, comprising the following steps: - generating a geometric model from the CAD model, comprising a sampling of a set of 3D points extracted from straight edges of said CAD model, so as to generate 3D surface points, - designating reference viewpoints in advance from a priori knowledge of the current viewpoints of the camera, - acquiring, by means of a camera, a plurality of images taken from viewpoints of interest of said object, - associating each acquired image with a reference viewpoint to form partitions and identifying a reference image for each partition, - selecting for each partition of the n 3D surface points extracted from the straight edges of the 3D model and having the 3D orientation of the straight edge from which they were extracted, comprising the following steps: - refining the camera poses using the geometric model and identifying the associations of 3D surface points with 2D contour points that correspond to the refined poses, - calculating a score for each 3D surface point based on the number of times that this 3D surface point is a match contour point, - projecting and clustering 3D surface points in each reference image, - selecting the n 3D surface points having the best score for each of the partitions.

2. The selection method according to claim 1, implemented in a convolutional neural network-based six-degree-of-freedom location, further comprising the steps of: - learning a first convolutional neural network to recognise a viewpoint of interest of the object among a set of reference viewpoints; - for each partition, learning heat maps corresponding to the 2D projections of the selected 3D surface points.

3. The selection method according to claim 2, characterised in that it further comprises a step for identifying, by means of the first convolutional neural network, the closest viewpoint among six predefined viewpoints calculated automatically as a function of a 3D bounding box of the 3D object and camera calibration settings.

4. The selection method according to claim 3, characterised in that it further comprises a partitioning sequence comprising the following steps: - annotating images for each learning sequence by launching 3D tracking, - associating the nearest reference viewpoint with each image based on 2D projections of 3D surface points and / or visibility tests, - identifying p reference images acquired from the viewpoint closest to p reference viewpoints, thus producing p partitions of the acquired images.

5. The selection method according to claim 4, characterised in that it also comprises a step of associating a partition with each image for the purpose of learning a multi-class detector.

6. The selection method according to any one of claims 2 to 5, characterised in that it also comprises a learning sequence during which: - a multi-class detector is learned, allowing detection of the industrial object and prediction of its distribution, - for each distribution, a keypoint detector is trained based on the location of heat maps.

7. A computer-implemented application of the selection method according to any one of the preceding claims for locating an industrial 3D object, a geometric model of which is generated offline, comprising the following steps: - refining a pose of a camera filming said object using the geometric model, this step being subject to prior initialisation, said refining comprising optimising the camera pose in order to maximise the number of correspondences between 2D contour points and 3D points, - validating the camera pose when obtaining a match ratio higher than a predetermined threshold, - generating a 3D location of said object, - constructing an appearance model of said object.

8. The application according to claim 7, characterised in that it also comprises the following steps carried out online: - constructing an appearance model from the pose thus refined, - extracting features in the following frames and matching with said appearance model, - estimating a pose for the 3D object, - determining a confidence score for said pose estimate.

9. The application according to claim 8, characterised in that the pose estimation step is followed by a step that is conditioned by the level of the confidence score in the following way: - for a confidence score lower than a first predetermined threshold value, performing a 3D relocation based on a set of appearance models already produced, - for a confidence score comprised between the first threshold value and a second predetermined value, refining, then validating the pose using the geometric model, - for a confidence score greater than the second predetermined value, confirming the 3D location.

10. The application according to claim 9, characterised in that the success or failure of the 3D relocation step leads respectively either to a new construction of an appearance model and a 3D location of the object, or to a re-initialisation of the pose validation and refinement step.

11. The application according to claim 10, characterised in that the initialisation or re-initialisation step prior to the execution of the pose validation and refinement step implements an initialisation module based on a convolutional neural network architecture.

12. An augmented reality system exploiting 3D objects located by implementing the surface point selection method according to any one of claims 1 to 6.