METHOD FOR PRODUCEING A DIGITAL 3D MODEL OF ONE OR MORE AIRCRAFT ELEMENTS FOR PRODUCEING AUGMENTED REALITY IMAGES

Description

DOMAINE TECHNIQE ET ETAT DE LA TECHNIQUE ANTÉRIEURE

[0001] The present invention relates to the field of data acquisition and processing for the implementation of a digital augmented reality system dedicated to the inspection of aircraft or aircraft components, for example to provide assistance during maintenance operations on an aircraft or on an aircraft component.

[0002] The inspection of the external surface condition of an aircraft is typically carried out by an operator with a limited field of vision.

[0003] To improve a device inspection procedure, one approach is to assist the operator performing the inspection by developing an augmented reality digital system. This system overlays one or more virtual elements onto a real image. These virtual elements take the form of one or more 3D objects embedded within a sequence of images and representing components of the device. Such 3D objects can typically be created from models generated by CAD (Computer-Aided Design) software and entirely designed using that software.

[0004] However, a 3D model of all the parts of the device in question is not necessarily available.

[0005] Furthermore, some 3D models produced by CAD tools may be confidential.

[0006] Finally, some 3D models from CAD tools may prove insufficient and / or not directly usable by a digital system allowing the implementation and projection of augmented reality images.

[0007] The method proposed in SVENJA KAHN: "Reducing the gap between Augmented Reality and 3D modeling with real-time depth imaging", VIRTUAL REALITY, vol. 17, no. 2, December 23, 2011 (2011-12-23), pages 111-123, ISSN: 1359-4338, DOI: 10.1007 / s10055-011-0203-0 D1 detects in real time the differences between a scene and a 3D model of that scene. Then, the detected geometric differences are used to update the 3D model, thus bridging the gap between augmented reality and 3D modeling. EXPOSÉ DE L'INVENTION

[0008] One object of the invention is to provide an improved method for creating at least one 3D digital model of at least a part of a component, or of at least one component or assembly of aircraft components, for the production of augmented reality image(s). The invention is defined by the independent claim. Other aspects are defined by the dependent claims. BRÈVE DESCRIPTION DES DESSINS

[0009] The present invention will be better understood upon reading the description of the given exemplary embodiments, provided for illustrative purposes only and in no way limiting the application, with reference to the accompanying drawings in which: [ Fig. 1 ] serves to illustrate an acquisition step using a scanner, in this example a non-contact type, of a point cloud of at least one aircraft component for the purpose of creating a 3D model of that component and using that 3D model in an augmented reality image; Fig. 2 ] is used to illustrate a scanner calibration step using a plate with a set of holes; [ Fig. 3A ] ] Fig. 3B ] are used to illustrate a point cloud obtained by scanning an aircraft component; [ Fig. 4 ] is used to illustrate a mesh of the part obtained from the aforementioned point cloud; [ Fig. 5 ] serves to illustrate a step in removing a triangular cell from the aforementioned mesh; [ Fig. 6 ] serves to illustrate a step of adding triangular cells to the aforementioned mesh; [ Fig. 7 ] serves to illustrate a repair or correction of the mesh by implementing a substep of surface joining; [ Fig. 8A ] ] Fig. 8B ] serve to illustrate a step in simplification of 3D meshing; [ Fig. 9 ] serves to illustrate a first method of selecting cells from a 3D mesh surface for the purpose of its segmentation; [ Fig. 10 ] serves to illustrate a second method of selecting cells from a 3D mesh surface for the purpose of slicing it; [ Fig. 11 ] ] Fig. 12 ] ] Fig. 13 ] are used to illustrate the creation of a selection area of ​​a particular shape to perform a 3D mesh clipping; [ Fig. 14A ] ] Fig. 14B ] are used to illustrate a reconstruction of the internal volume of a part model; [ Fig. 15 ] ] Fig. 16 ] serve to illustrate a step in reconstructing the internal volume of a part model; [ Fig. 17A ] ] Fig. 17B ] serve to illustrate another step in reconstructing the internal volume of a part model; [ Fig. 18 ] is used to illustrate a 3D model of an aircraft part inserted into an augmented reality image, which was obtained by scanning and then digitally processing at least one mesh resulting from this scan; Fig. 19A ] ] Fig. 19B ] ] Fig. 19C ] ] Fig. 20A ] ] Fig. 20B ] ] Fig. 20C ] serve to illustrate the transformation of a 3D model of an aircraft part obtained by scanning into a 3D model with reconstructed internal geometry for use in augmented reality images°; Identical, similar or equivalent parts of the different figures carry the same numerical references in order to facilitate the transition from one figure to another.

[0010] The different parts represented in the figures are not necessarily shown on a uniform scale, in order to make the figures more legible. EXPOSÉ DÉTAILLÉ DE MODES DE RÉALISATION PARTICULIERS

[0011] We refer first to the figure 1 This document illustrates a "digitization" step of an aircraft part 10, and in particular a 3D model usable in augmented reality. For the sake of simplicity, this description will refer to an aircraft part 10, but the described processes can be applied more generally to an element corresponding to a portion of an aircraft part, or to a set of aircraft parts, for the purpose of producing a 3D model of that portion, part, or set of parts. This 3D model is intended to be displayed in an augmented reality image and can also be used to establish an object tracking model for the production of augmented reality images.

[0012] The digitization includes a step of exposing the part 10 using a three-dimensional scanner 20, typically an optical scanner emitting light radiation, for example blue light, in particular narrowband, can be used to perform such a digitization of the shape of the part 10. According to a particular example, a scanner from the ATOS Core range manufactured by the company GOM, in particular an ATOS Core 5M MV300 scanner can be used.

[0013] This step contributes to the creation of a 3D model quickly while avoiding the use of saved graphic objects entirely created from CAD software.

[0014] The 3D scanner 20 produces in particular a set of digitized points of said element in a 3-dimensional space, these points being representative of the shape of the part 10. This set of points is obtained by projecting a pattern of light onto the surface of the part 10 using at least one light source.

[0015] The internal light source can be, for example, one or more lasers, such as vertical cavity surface emission lasers (VCSELs). The light source typically projects the light radiation stroboscopically or pulsed at a specific frequency. One or more images of the structured light projected onto the surface of part 10 are acquired using at least one optical sensor, for example, one or more cameras and / or imagers. The digitized point cloud is obtained from this image or these images.

[0016] The three-dimensional scanner 20 is here connected to a data processing unit 30, typically a computer, equipped in particular with at least one data processing processor and one or more memories.

[0017] The scanner 20 can be positioned or fixed on a support 26, for example an articulated arm, movable with several degrees of freedom. In the illustrated embodiment, the aircraft part 10 to be scanned is itself arranged on a movable support 24.

[0018] Optionally, a preliminary calibration step (also referred to by the English term "calibration") of the scanner 20 can be performed. In the example illustrated on the figure 2 This step is performed using a plate 40 with patterns having a known regular distribution. Typically, the plate 40 is perforated and the calibration patterns are holes.

[0019] A surface preparation step for the part 10 to be scanned can also be carried out to improve the scan quality. Such a step might consist, for example, of coating the surface of the part 10 with a light-absorbing coating and / or a coating that modifies its color. In one particular embodiment, a coating based on MR®< 2000 Anti-Reflex L could be used. To allow for better positioning of the points obtained during image acquisition, the part 10 can be fitted with markers 12a, 12b, 12c on its surface, for example, in the form of stickers or adhesive labels affixed at three distinct and non-aligned points on the part 10.

[0020] THE figures 3A-3B These points are used to illustrate an image I 10 of a scanned aircraft part 10 with three reference points 51a, 51b, 51c corresponding to the respective positions of the stickers or labels. This positioning on part 10 is known, and the positioning of new points 52a, 52b is deduced using these reference points 51a, 51b, 51c. The acquisition and determination of the coordinates of new points is then carried out iteratively by registration with respect to a cloud of scanned points whose coordinates have themselves been determined from points 51a, 51b, 51c, 52a, 52b.

[0021] An analysis of the scanned points can then be performed using an application, specifically a data processing software tool that can be installed, for example, on the computer connected to the scanner or accessed via that computer. This software tool allows for visual and interactive exploration, as well as graphical representation and processing of imported measurement data. For example, such a tool could be "ATOS Gom Scan". The resulting 3D point file can then be corrected by removing any errors or extraneous elements recorded during the scanning process.

[0022] For example, when the scanned part 10 is placed against a surface such as a protective mat, unwanted points representing that surface are removed. This can be achieved by defining a selection area within the aforementioned tool, allowing the user to select these unwanted points and then, using a function within the tool, removing them.

[0023] A corrected 3D point cloud file can then be processed to form an M10 mesh. This M10 mesh is typically composed of points and edges forming cells in the form of polygons, usually triangles or quadrilaterals. The cells themselves constitute polyhedra, for example, trihedrons. Such an M10 mesh differs from a parametric model obtained using CAD software, which contains a set of functions and parameters defining volumes.

[0024] Such an M 10 mesh can be obtained in particular using a processing which uses for example a meshing algorithm which notably implements a Delaunay triangulation type operation.

[0025] The M10 mesh can be generated using a second software tool. This second tool can perform analysis, visual and interactive exploration, graphical representation of imported measurement data, and processing of this data. Such a tool, accessible via the processing unit 30 or computer 30 and possibly stored there, can be used, in particular, to perform shape and dimension analysis, as well as 3D mesh processing. In one specific implementation, this second software tool could be the "GOM Inspect Suite," especially when scanning is performed using a GOM scanner. 3D modeling and rendering software such as "Blender" or 3D model design software such as CATIA® can also be used.

[0026] A graphical representation of the M10 mesh of the scanned part obtained using the second tool is given, for example, on the figure 4 In this figure, the meshes or cells composing the mesh are not visible due to their density.

[0027] In this particular embodiment, the second tool is configured to count the geometric elements appearing on the graphical representation of part 10. Here, for example, a number of distinct holes in the mesh M10 and its component cells is counted and can be compared to the number of holes actually present on the digitized part 10. This functionality can help in repairing the mesh M10. It allows the identification of holes 61 and 62 corresponding to incomplete portions of the mesh M10.

[0028] We can then perform the repair of the M 10 mesh. Such a repair step can be carried out using the second tool mentioned above and include the removal of certain cells, for example outliers resulting from an inaccurate positioning of one or more points during the digitization step.

[0029] In the example of implementation illustrated on the figure 5 where the mesh is formed of triangular cells, a step of removing an aberrant triangle 71 located inside a selection area 72 is implemented.

[0030] Hole interpolation, to modify the outline of one or more holes or to remove one or more holes, can also be performed, particularly using the second tool mentioned above. In the example illustrated on the figure 6 A hole (81) that we want to remove is shown. In this case, we need to add cells to make the correction.

[0031] Mesh repair can also include, particularly with the second tool mentioned above, a step of joining surfaces together. For example, when the surfaces to be joined are portions of walls around a hole, rather than removing the hole itself using an interpolation step as described earlier, these separate portions can be joined into a single surface.

[0032] In the example shown on the figure 7 Such a joining of surfaces results in the formation of a bridging surface 92 connecting portions 91, 93 of the walls of a hole. This operation can be performed, for example, by selecting at least two cells to be joined, here in the form of two disjoint triangles, and using a bridging function of the aforementioned tool. This operation can be repeated several times to create the bridging surface 92.

[0033] At this stage, it may be desirable to reconstruct one or more new useful surfaces to complete the M10 mesh, specifically corresponding to areas located within the internal volume of part 10 that are not visible, i.e., located outside the scanner's field of view, even when the scanner is moved relative to part 10. Such areas are therefore not included in the set of points or the point cloud digitized by the scanner mentioned previously. Alternatively, it may be possible to reconstruct only a portion of the part's geometry and prioritize only one or more so-called "useful" surfaces, for example, surfaces that, in an augmented reality image, are represented to facilitate disassembly or highlight specific areas.For example, we may want to highlight on a 3D model areas to be painted or masked of part 10 during a painting process step carried out by an operator using an augmented reality image stream.

[0034] Prior to such a reconstruction step, the mesh can be simplified to reduce the number of elements, particularly the number of cells it contains. To perform this process, the mesh resulting from the previous repair substep(s) can then be exported to a third data processing tool. For example, CAD or 3D modeling software such as CATIA® can be used, with the mesh then exported as an .STL file. Alternatively, this step can be performed using the first and / or second tool mentioned above. For example, 3D modeling and rendering software such as Blender, developed by the Blender Foundation, or GOM Inspect Suite can be used for this simplification.

[0035] Such simplification, where the number of cells—for example, the number of triangles forming the mesh—is reduced while preserving the overall geometric shape of the mesh, is typically performed to reduce the file size(s). This reduction is all the more useful given the significant processing required for rendering and real-time calculations to implement an augmented reality image. For example, the number of cells, particularly triangles forming the mesh, can be reduced to a specific limit set by the tool's user, which could be, for instance, tens of thousands of cells, such as 50,000 triangles. This limit is chosen to maintain the mesh's shape fidelity to the actual part 10 and therefore depends on the complexity and size of that part 10.Such a step can be implemented using a specific feature of the third software tool called "decimate," which allows you to select the mesh and choose the desired number of cells (triangles). This feature and / or simplification step can be implemented using a polygonal simplification algorithm.

[0036] Such a simplification, with here a reduction in the number of triangles, is illustrated on the figures 8A-8B representing respectively a mesh M 10 of a portion of part 10 before polygonal simplification and a new mesh M' 10 of the simplified part obtained at the end of a polygonal simplification step.

[0037] The reduction achieved and the level of simplification implemented can be adapted according to the computing capabilities of computer resources which are intended to be used as a production support for an augmented reality image stream and into which a graphic representation of the part obtained from the mesh is intended to be inserted.

[0038] A division of the M10 or M'10 mesh, also called "slicing", aimed at separating it into several parts can also be carried out.

[0039] Such a step can be performed, for example, to isolate and possibly reconstruct a portion of the mesh corresponding to a particular surface of the scanned object 10, such as a bore, in order to facilitate subsequent processing or to highlight it through a specific display. The isolated portion can be duplicated, and a copy can then be processed as an independent mesh.

[0040] Such processing and display can be achieved, for example, through a module of the third software tool, for example, the "Catia Composer" module.

[0041] The segmentation is carried out by first selecting cells, in particular triangles, in order to separate the mesh into at least two distinct meshes.

[0042] During this step, the initial mesh can be retained, and two new meshes can be created, each representing one of the two parts of the initial mesh, selected or not. The initial mesh can be used for object recognition or tracking to implement an augmented reality image sequence.

[0043] A possible new cutting operation can then be carried out on one of these meshes.

[0044] Selecting and deleting cells to perform this segmentation can be done as shown in the example illustrated on the figure 9 , with a selection area 101, smaller than the surface 102 to be processed, which is moved. According to another embodiment illustrated on the figure 10 , we can also make this selection with a selection area 101' called "trapdoor" of size and shape corresponding to that of a surface 102' to be treated, in other words that we want to cut out.

[0045] In the example of implementation illustrated on the figures 11 et 12 To allow for more precise localization of the 101" trapdoor-type selection area, or when the shape of the selection area is difficult for a user to create manually, this selection area can be created by first placing several preference points, at least three non-aligned points P1, P2, P3, on a portion of the mesh to be selected. A plane PL is then defined from these three points P1, P2, P3.

[0046] A feature such as the one called "Part Design" in the aforementioned tool can then be used to create a selection area according to the desired geometry. In the example shown on the figure 12 The 101" selection area is in the form of a ring. On the figure 13 , the selection zone 101" corresponds this time to a non-planar zone formed of distinct and non-parallel PL, PL' planes.

[0047] Other types of selection areas besides those described and illustrated previously can be implemented. In particular, it is possible to create selection areas with a curved outline.

[0048] It is also possible at this stage, or starting from the simplified M'10 mesh obtained previously, to perform one or more corrections, and in particular to delete mesh element(s) such as at least one point, at least one edge, or at least one cell, for example, in the form of a triangle. Such a step can correct any errors in the original mesh or any errors introduced during the previously mentioned cutting step.

[0049] It is also possible to make one or more corrections to the simplified M' 10 mesh or to one of those obtained after cutting this time by adding one or more mesh elements for example by adding cells, in particular triangular cells.

[0050] Optionally, it is also possible to merge several distinct meshes. For example, to a mesh representing part 10 or the complete element 10, at least one additional mesh can be added or joined; for example, a mesh representing a bore and / or a chromed area or a shoulder can be joined to a mesh representing a part containing this bore, this chromed area or this shoulder.

[0051] Another example of mesh merging involves combining all parts of a component that share similar shapes. For instance, this can create a resulting mesh in which all the holes in a part are represented. Other combinations of shapes, such as bores, chamfers, bosses, teeth, notches, shoulders, recesses, slots, grooves, ribs, and projections, can also be achieved.

[0052] The additional mesh(s) may have different graphical representations than the original mesh, for example different colors to highlight them.

[0053] A so-called "reconstruction" step can then be carried out. This step corresponds to adding to a mesh obtained through one or more of the processing steps described above one or more portions corresponding to parts of the scanned part 10 that are not located in the field of view of the scanner during the digitization step, and in particular internal geometric elements of part 10.

[0054] This step can be performed using a third software tool, specifically a CAD tool. For example, CATIA® and its "Part Design" module can be used for 3D model design. Other CAD tools such as SolidWorks, or 3D modeling tools such as Blender, 3ds Max, or even Rhinoceros 3D, can also be used for this type of reconstruction. The reconstruction can be carried out using a drawing function in one of the aforementioned tools to add one or more missing patterns.

[0055] Prior to such a reconstruction, a work plane can be defined in the CAD tool. This definition is achieved by selecting three distinct and non-aligned points on the mesh of part 10 in a format usable by the CAD tool.

[0056] In some cases, as in the example of implementation illustrated on the figures 14A-14B (giving respectively an internal view and an external view of a part 110), the realization of a very precise reconstruction may be optional and one may wish to recreate a general appearance of the internal volume 111 of a part 110.

[0057] In one case, in particular, where a definition drawing is not available but the geometry of the part is known, and in particular where a phase plane is available, a reconstruction can be carried out using point projections of a mesh and shaping operations.

[0058] In the examples illustrated on the figures 15 et 16 we perform a projection of a zone 121, 121' of a mesh M 100 , M' 100 obtained using processing as described above, in order to add an additional portion of mesh to this mesh and complete it.

[0059] Another type of reconstruction illustrated on the figures 17A-17B This involves modifying a mesh M" 100 obtained after steps as described above by locally removing certain cells and adding others, so as to create a hole 131 that extends within its internal volume. It sometimes happens that at this stage the resulting mesh contains errors that one wishes to correct. For example, a hole in a scanned part may have been completely closed due to a scanning error or incorrect interpolation.

[0060] A 3D model produced from a mesh obtained using the processing described above, when integrated into a virtual reality image, is caused to move and / or have a shape, a dimension, a positioning that are likely to evolve according to the positioning and the distance from which a user of an augmented reality image projection system is located of room 10. In a sequence of augmented reality images to allow a correct display of the surfaces of room 10 via a software platform such as "Diota Player", one solution consists of using copies of one or the other of the meshes described above having undergone a shifting step.

[0061] A mesh shift can be implemented by translating each of the mesh polygons along a direction parallel to a normal to the surface of the polygons. This type of shift differs from a homothety in that the goal here is to preserve the internal diameters.

[0062] Such a displacement can be achieved, for example, using the "mesh offset" feature of the "Part Design" tool. With this tool, an offset parameter is set, for example, according to an offset increment that can be between 0.2 and 0.5 mm. Here, it is preferable to offset a mesh representing element 10 or the entire part 10, rather than offsetting meshes corresponding to individual surfaces of part 10.

[0063] According to one embodiment, when the part or element 10 to be modeled has a corresponding symmetrical element or part in the aircraft, only one of the two parts can be exposed to the 3D scanner, and the aforementioned processing(s) can be performed on the point cloud obtained during this scan. From this point cloud, a 3D model of the symmetrical part can be created using a symmetrical model creation function of one or more of the tools mentioned previously. In particular, a symmetry operation of the Catia "Part Design" software tool can be used. First, a plane of symmetry is created, for example, by defining three distinct points. Then, the part or an area of ​​the part whose symmetrical counterpart is to be created is selected.

[0064] A 3D model obtained through the steps described above can then be imported into an augmented reality image display tool. Such a tool could be, for example, "Diota Connect for Composer" from Diota. In this case, the created 3D model can be imported in a CatPart format.

[0065] On the figure 18 An augmented reality image 181 provides a view of such a 3D model 182 of the object 10, obtained by scanning and then a series of processing steps as described previously. The final result is close to that which would be obtained using a model entirely designed by CAD.

[0066] THE figures 19A-19C respectively show an M20 mesh obtained after scanning another aircraft part, while the figures 20A-20CThey show a 3D model M' 20 of this part obtained after a reconstruction step of the internal volume of part 20, created from the mesh M 20, here performed using CAD software. This recreates one or more internal surfaces that a simple scan alone cannot produce.

[0067] A virtual object superimposed on an augmented reality image stream can be produced using a hybrid 3D model composed of a geometric 3D model, as described previously, and a parametric model. The superimposition is achieved using an object tracking software tool, also known as an object tracking engine, which employs object tracking algorithms. Such algorithms typically include steps for predicting the object's position based on information about its position and movement, and then mapping the object within an image area using the parametric 3D model, which incorporates the object's displacements and deformations and is iteratively updated.

Claims

1. Method for creating a 3D digital model of at least one aircraft element (10) in order to produce augmented reality image(s), the method comprising steps consisting in: a) exposing to a scanner (20) at least one element (10) of an aircraft so as to produce a set of digitised points representative of the 3-dimensional shape of said element (10), then, using a computer (30) and / or at least one computer processing unit (30): b) forming, from said set of digitised points, at least one given 3D mesh representative of the shape of said element (10), c) modifying a replica of said given mesh so as to add at least one missing pattern to said replica and to produce a 3D model of said aircraft element, the method further comprising at least one step of cutting a first mesh corresponding to said given mesh or to a duplicate of said given mesh, the cutting being performed using said computer (30) or said computer processing unit (30) and comprising the sub-steps of: - defining a selection area (101, 101', 101") including a plurality of polygon cells of the first mesh, - deleting the polygon cells located in said selection area so as to form a second mesh, - defining a third mesh resulting from the difference between the first mesh and the second mesh, wherein the replica of said mesh results in adjoining the third mesh to another mesh, this other mesh being obtained by cutting the first mesh according to a second selection area, said selection area and said second selection area respectively corresponding to portions of said element (10) having similar shapes.

2. Method according to claim 1, wherein the missing pattern corresponds to a pattern of said element (10) located in its inner volume and / or which is located outside the field of view of the scanner during the exposure of the scanner in step a).

3. Method according to one of claims 1 or 2, wherein said replica of said given mesh is obtained using at least one step of reducing, carried out by said computer (30) or said computer processing unit (30), a number of polygon cells constituting said given mesh (M10) while conserving an overall geometric shape of said given mesh.

4. Method according to one of claims 1 to 3, wherein the replica of said given mesh is obtained by performing a correction step using said computer or said computer processing unit (30), by: - adding one or more polygon cells to said given mesh and / or, - removing one or more polygon cells from said given mesh, and / or - moving one or more polygon cells of said given mesh.

5. Method for producing augmented reality images of an aircraft element (10), comprising: - forming a 3D model of said aircraft element using a method according to one of claims 1 to 4, - displaying said 3D model in a sequence of augmented reality images.

6. Method for producing augmented reality images of an aircraft element (10) according to claim 5, wherein the sequence of augmented reality images is formed using a 3D tracking tool using a parametric model translating movements and deformations of said given 3D mesh or of said replica.