Method, module and system for projecting an image calculated based on a digital model onto a workpiece

Abstract

A method for projecting an image calculated based on a digital model (31) associated with a real workpiece (4) recorded on a digital information server (3) onto the real workpiece in order to view the workpiece in augmented reality, the method comprising the steps of capturing an image of the real workpiece by means of a camera device, calibrating in real time a reference system associated with the digital model with a reference system of a video capture system and a reference system of the real workpiece, the method comprising a step of reprocessing the calculated image according to the topology of the digital model and according to the orientation of the projection device relative to the real workpiece.

Description

Technical Field

[0001] The invention relates to a system for visual display of information about real objects for augmented reality applications in industry, in particular for the design, manufacture, assembly or maintenance of technical artifacts.

[0002] Various augmented reality systems are known for extending the perception of visual reality with information originating from digital models and more generally digital content that can be projected on the surface of a real object, for example icons or texts specifying the parts of the object with which an operator must intervene.

[0003] These augmented reality solutions make it possible to increase the efficiency of manual work steps in manufacturing, assembly and maintenance, and at the same time improve the quality of the work. By using models and other dispensable measuring instruments, the precise transmission of information, such as the digital planning state (CAD model) directly to the workpiece, complicates the transmission and makes the manufacturing plan error-prone. Visual difference comparisons can be performed at any time in a way that is intuitive to the user. In addition, work instructions, such as stepping instructions, can be provided directly on the work object or in the user's field of vision, i.e. exactly where they are really needed. Typical application scenarios of the present invention include worker assistance systems for displaying assembly and maintenance instructions and information for quality assurance. For example, assembly locations or drill holes can be accurately marked, or welding points or support points to be verified can be identified. The system can also provide assistance to on-site maintenance personnel by non-resident experts who can remotely monitor the projection with the help of an integrated camera device.

[0004] The workpiece to be machined must be precisely calibrated with respect to the position and orientation of the projector in order to allow a correlation between the reference of the additional information and the reference of the real object.

[0005] The solution for calibrating the reference system is based on the use of markers: A system for visual display of information about a real object comprises a projection unit, which graphically transmits an item of information to the object, and a dynamic tracking device with a 3D sensor system, which determines and maintains the position and / or orientation of the object and / or the projection unit in space. The control device of the projection unit adapts the transmission of information to the current position and / or orientation of the object and / or the projection unit determined by the tracking device.

[0006] The combination of a projector with a dynamic 3D tracking device allows a continuous and automatic calibration (dynamic referencing) of the projector and / or the object on which an item of information must be displayed relative to the working environment. It follows that both the projection unit and the object can be freely displaced, since the graphic and / or pictorial transmission of the information is automatically tracked at each movement of the projection unit or the object.

[0007] Other solutions make it possible to calculate the position of real objects relative to the user's point of view, so that digital information is permanently aligned with the real world, without the need to stick markers. SLAM (Simultaneous Localization and Mapping) technology makes it possible to reconstruct the environment in 3D.

[0008] Other solutions are based on automatic learning algorithms that allow objects to be recognized and tracked over time. Background Art

[0009] From the prior art, the international patent application WO201591291 is known, which relates to a method for executing and monitoring the steps of machining on a workpiece using a system comprising a projector, a photogrammetric device and a control unit of the relative position of the projector and the photogrammetric device. The method comprises the steps of: referencing for the purpose of determining the position of the workpiece relative to the projector and the photogrammetric device in a predetermined coordinate system; projecting work instructions on the workpiece; machining the workpiece by an operator; and monitoring the machining by scanning at least a part of the surface of the workpiece. All steps are performed by the system at the same workstation within the framework of an integrated process, with the exception of the machining performed by the operator.

[0010] International patent application WO200493444 is also known, which relates to a method for making it possible to display output images on an object. To this end, a set of unique markings is fixed to the object at predetermined positions. An input image of the object and the markings is captured by a camera device in a fixed physical relationship relative to a projector. The position of the projector relative to the markings is determined from the image. Then, depending on the position of the projector and the unique markings, one or more output images related to the object can be projected onto the object at predetermined positions.

[0011] French patent FR3021784 describes a method for projecting information derived from a digital design model, the method comprising the following steps:

[0012] - a calibration step comprising acquiring characteristic data originating from the surface of interest, comparing said characteristic data with digital data originating from the digital design model, and determining the spatial position of a projection device comprising a video projector and at least two different image acquisition devices; and

[0013] - A step of projecting information originating from a digital design model onto said surface of interest by means of said video projector according to said spatial position.

[0014] International patent application WO200332129 describes another example of a system that makes it possible to view deviations on a real surface relative to a nominal or mapped surface, using a system and method comprising: mapping the spatial coordinates (e.g. x, y and z) of the real surface on a computer; comparing the mapped real surface with the nominal surface to achieve a three-dimensional distribution of deviation values ​​(D); processing the distribution in the form of a topographic map of a plurality of contours or areas, each contour or area having the same or substantially the same deviation value (D); and optically projecting the topographic map on the real surface according to an initial mapping of the surface to produce a display of the surface deviations (D) directly on the real surface. These deviations are measured according to a direction D perpendicular to the real surface, which makes it possible to give a three-dimensional distribution in coordinates x, y, D. The optical projection is preferably a laser projection. The mapping and projection on the real surface are performed and coordinated relative to three reference points on the surface.

[0015] US patent application US20160358382 describes an example of a method for real-time augmented reality using 3D projection technology, the method comprising:

[0016] - sensing one or more scans of physical objects in the scene via a 3D sensor coupled to the computing device;

[0017] - generating, by means of a computing device, one or more 3D models of the physical object based on the database or the scan results;

[0018] - determining, by means of a computing device, the position of one or more 3D models relative to a projector in the scene;

[0019] - prearranging, by a computing device, image content based on the position of one or more 3D models to generate a map of the rendered image and a calibration result; and

[0020] - Using a projector coupled to the computing device, superimposing a map of the rendered image onto physical objects in the scene using the calibration results.

[0021] US patent application US20170054954 describes a system for visually displaying information about a real object, the system comprising:

[0022] - a projection unit comprising at least one projector, said at least one projector transmitting geographically or visually an item of information to the object, said information comprising at least information originating from a manufacturing plan of the object;

[0023] a dynamic tracking device with a 3D sensor system comprising at least one camera device, wherein the dynamic tracking device determines and maintains the trajectory and / or orientation of the object and / or the projection unit in space;

[0024] a plurality of markers arranged at reference points of the environment in which the system is used and wherein these markers are detectable by a 3D sensor system of the dynamic tracking device; and

[0025] wherein the projection unit is controlled to adapt to the transmission of information on the current position and / or orientation of the object and / or the projection unit determined by the dynamic tracking device, and wherein the dynamic tracking device is configured to continuously detect the position and / or orientation of the object and / or the projection unit in real time so that a worker can identify any deviations from the manufacturing plan based on the transmission of the projection unit.

[0026] Disadvantages of existing technology

[0027] The solutions of the prior art are not completely satisfactory, because in some cases the projection of information in augmented reality disturbs the operator. For example, when the real workpiece has a through cavity and the projector is on the axis of such a cavity, the light beam may dazzle the operator when he intervenes on the side of the workpiece opposite to the projector.

[0028] Likewise, when projecting on a curved portion of a workpiece on which the augmented reality image is projected or the augmented reality image is offset relative to a reference plane perpendicular to the projection axis, the information will become unusable due to the resulting deformation or blurring. Summary of the invention

[0029] The present invention aims to overcome these drawbacks by proposing an automatic adaptation of the projector according to the topology of the workpiece and the position relative to the projector.

[0030] To this end, according to a first aspect of the present invention, a projection method for projecting an image calculated based on a digital model recorded on a server onto a real workpiece associated with the digital model to view the workpiece in augmented reality is proposed, the method comprising the following steps:

[0031] - Capture images of real workpieces through camera devices,

[0032] - Calibration in real time of the reference frame associated with the digital model with the reference frame of the video capture system and with the reference frame of the real workpiece.

[0033] According to a first aspect of the invention, the method comprises a step of reprocessing the calculated image according to the topology of the digital model and according to the orientation of the projection device relative to the real workpiece.

[0034] The reprocessing step may include detecting through-regions of the real object based on the topology of the digital model and based on the orientation of the projection device relative to the real workpiece, and may include attenuating the light intensity of regions of the calculated image corresponding to the detected through-regions.

[0035] Advantageously, the reprocessing step may comprise detecting an area of ​​the digital model that is offset in depth relative to a reference plane perpendicular to an axis passing through the focus of the projector and the centre of gravity of the real workpiece, and may comprise modifying the calculated image in the portion corresponding to the area.

[0036] According to one option, the modification may comprise masking the additional digital information in the calculated image.

[0037] According to another option, which may be combined with the first option, the modification may comprise digitally correcting the pixels of said area to compensate for an offset between the projection surface and said reference plane.

[0038] According to a particular embodiment, the correction may comprise: generating an error matrix formed by the difference between a matrix calculated based on an image acquired based on a real workpiece and a reference matrix corresponding to the topology of the digital model repositioned in the same reference system; and then generating an error criterion and modifying the elements of the reference matrix of the image to be projected during the adaptation process so as to minimize the error matrix and / or its scalar measurement.

[0039] According to a second aspect of the present invention, a projection module is provided for projecting an image calculated based on a digital model recorded on a server onto a real workpiece associated with the digital model so as to view the workpiece in augmented reality, the projection module comprising:

[0040] - Image capture module for real artifacts,

[0041] - a calibration module configured to calibrate in real time a reference frame associated with the digital model with a reference frame of the video capture system and with a reference frame of the real workpiece.

[0042] According to the invention, the projection module comprises a post-processing module which is designed to post-process the calculated image as a function of the topology of the digital model and as a function of the orientation of the projection device relative to the real workpiece.

[0043] The various modules may be implemented in electronic form, in software form or even, for example, in FPGA form.

[0044] According to a third aspect of the present invention, a projection system is provided, comprising:

[0045] - Camera device;

[0046] - digital information servers including digital models;

[0047] - Real artifacts associated with the digital model;

[0048] - One or more improved projection modules according to the second aspect of the invention or its improvements.

[0049] According to one option, the reprocessing module is configured to implement one or more improved reprocessing steps according to the first aspect of the invention or its improvements.

[0050] According to a fourth aspect of the invention, a computer program product is proposed, which can be directly loaded into the memory of a computer, the computer program product comprising software code portions for performing the steps of the method according to any of the preceding claims when said program is executed on a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Other data, characteristics and advantages of the invention will emerge from reading the description of non-limiting implementations and embodiments, with reference to the accompanying drawings, in which:

[0052] - Figure 1 shows an embodiment of the system according to the invention,

[0053] - Figure 2 A first implementation of the method according to a first embodiment of the method of the invention is shown,

[0054] - Figure 3 A second implementation of the method according to the second embodiment of the method according to the invention is shown. DETAILED DESCRIPTION

[0055] Since the embodiments described below are by no means limiting, it is in particular possible to envisage variants of the invention which comprise only a selection of the features described below, separated from the other features described, if this selection is sufficient to provide a technical advantage or to distinguish the invention from the prior art. This selection comprises at least one feature, which is preferably functional and without structural details, or with some structural details, if this part alone is sufficient to provide a technical advantage or to distinguish the invention from the prior art.

[0056] Figure 1 A first embodiment of a projection system 1 according to the invention is shown.

[0057] The projection system 1 comprises:

[0058] - Camera device 2;

[0059] - a digital information server 3 comprising a digital model 31;

[0060] - a real workpiece 4 associated with the digital model 31;

[0061] A projection module 5 according to an embodiment of a projection module of the present invention.

[0062] according to Figure 1 In the example shown, the camera device 2 and the projection module 5 are integrated into the same device 6 .

[0063] The projection module 5 is configured to project an image onto the real workpiece 4 .

[0064] The projection image is calculated based on the digital model 31 recorded on the digital information server 3.

[0065] Therefore, according to Figure 1 In the embodiment shown, the projection module 5 comprises:

[0066] a module for capturing an image of the real workpiece, which is formed here by a camera device 2;

[0067] -Computing unit 7.

[0068] The computing unit 7 is configured to calibrate in real time the reference system associated with the digital model 31 with the reference system of the capture module 2 and with the reference system of the real workpiece 4 .

[0069] Furthermore, the computing unit 7 is configured to reprocess the calculated image as a function of the topology of the digital model 31 and as a function of the orientation of the projection device 5 relative to the real workpiece 4 .

[0070] Figure 2 The implementation of a first embodiment of a method P1 according to the invention is shown.

[0071] Method P1 includes:

[0072] - A step of capturing an image of a real workpiece 4 by means of a camera device 2;

[0073] - a step of real-time calibration of the reference system associated with the digital model 31 with the reference system of the camera device 2 and with the reference system of the real workpiece 4;

[0074] A step of reprocessing the calculated image according to the topology of the digital model 31 and according to the orientation of the projection device relative to the real workpiece 4 .

[0075] like Figure 2 As shown, the reprocessing step of method P1 also includes:

[0076] - detecting two through-regions 41 and 42 of the real object 4 based on the topology of the digital model 31 and the orientation of the projection module 5 relative to said real workpiece; and

[0077] - Attenuating the light intensity of the regions of the calculated image corresponding to the two detected through regions 41 and 42 .

[0078] Furthermore, this makes it possible to avoid dazzling the eyes of a user receiving a light beam which is projected through the corresponding area and which originates from the projection of the reprocessed image by the projection module 5 .

[0079] According to the second embodiment of the method of the present invention, method P2 is proposed, and only the differences compared with method P2 are described.

[0080] Figure 3 An implementation of the second embodiment is shown.

[0081] The reprocessing steps of method P2 include:

[0082] - detecting an area of ​​the digital model 31 which is offset in depth relative to a reference plane perpendicular to an axis passing through the focal point of the projector and the centre of gravity of the real workpiece; and

[0083] - modifying the calculated image in the portion corresponding to said area.

[0084] The region of the real workpiece 4 which corresponds to the region of the digital model 31 which is offset in depth is denoted by the reference numeral 43 in the figure.

[0085] A reference plane perpendicular to the axis passing through the focal point of the projector and the center of gravity of the real workpiece is denoted by the number 44 in the figure.

[0086] The calculated image may include additional digital information 45 and 46, for example to indicate the diameter of the two top holes T1 , T2 shown in the figure.

[0087] The calculated image may also include additional digital information corresponding to the diameter of the bottom hole T3 shown in the figure.

[0088] However, in the embodiment shown in the figures, after detecting the offset between the region of the digital model 31 and the reference plane, the calculated image is modified by masking the additional digital information corresponding to the hole T3 in the calculated image.

[0089] According to a first variant of method P2 , described only with regard to its differences compared with method P2 , the modification may comprise a digital correction to compensate for the offset between the projection surface and the reference plane.

[0090] According to a second variant of method P2, described only with regard to its differences compared with method P2, which second variant can optionally be combined with the first variant, the correction step consists in: generating an error matrix formed by the differences between a matrix calculated from images acquired based on a real workpiece and a reference matrix corresponding to the topology of the digital model repositioned in the same reference system; and then generating an error criterion and modifying the elements of the reference matrix of the image to be projected during the adaptation process so as to minimize the error matrix and / or its scalar measure.

[0091] Of course, the present invention is not limited to the above described embodiments, and many improvements may be made to the embodiments without departing from the scope of the present invention. In addition, the various features, types, variations and embodiments of the present invention may be associated with each other according to various combinations, as long as they are not mutually incompatible or mutually exclusive.

Claims

1. A projection method for projecting an image calculated based on a digital model (31) associated with a real workpiece (4) recorded on a digital information server (3) onto the real workpiece to view the workpiece in augmented reality, the projection method comprising the following steps: Capturing an image of the real workpiece by a camera device; calibrating in real time a reference frame associated with the digital model with a reference frame of a video capture system and a reference frame of the real workpiece, The projection method further comprises a step of reprocessing the calculated image according to the topology of the digital model and according to the orientation of the projection device relative to the real workpiece, and Wherein, the reprocessing step further comprises: detecting a through-region (41, 42) of the real workpiece (4) based on the topology of the digital model (31) and the orientation of the projection device relative to the real workpiece, and The light intensity of a region of the calculated image corresponding to the detected through region is attenuated.

2. The projection method according to claim 1, wherein: The reprocessing step comprises detecting a region of the digital model (31) which is offset in depth relative to a reference plane (44) perpendicular to an axis passing through the focus of the projection device and the centre of gravity of the real workpiece (4); and modifying the portion of the calculated image corresponding to the region.

3. The projection method according to claim 2, wherein: The modifying comprises masking additional digital information in the calculated image.

4. The projection method according to claim 2, wherein: The modification includes digitally correcting pixels of the region to compensate for an offset between the projection surface and the reference plane.

5. The projection method according to claim 4, wherein: The correction consists in: generating an error matrix formed by the difference between a matrix calculated from the images acquired based on the real workpiece and a reference matrix corresponding to the topology of the digital model repositioned in the same reference system; and then generating an error criterion and modifying the elements of the reference matrix of the image to be projected in the adaptation process so as to minimize the error matrix and / or a scalar measure of the error matrix.

6. A projection module for projecting an image calculated based on a digital model (31) associated with a real workpiece (4) recorded on a digital information server (3) onto the real workpiece to view the workpiece in augmented reality, the projection module comprising: a module for capturing an image of said real workpiece, a calibration module configured to calibrate in real time a reference system associated with the digital model with a reference system of the module for capturing an image of the real workpiece and with a reference system of the real workpiece, wherein the projection module further comprises a reprocessing module configured to reprocess the calculated image according to the topology of the digital model and according to the orientation of the projection module relative to the real workpiece, and Wherein, the reprocessing module is further configured to: detecting through-regions (41, 42) of the real workpiece (4) based on the topology of the digital model (31) and the orientation of the projection module relative to the real workpiece, and The light intensity of a region of the calculated image corresponding to the detected through region is attenuated.

7. A projection system (1), comprising: Camera device; A digital information server (3) including a digital model (31); a real artifact (4) associated with the digital model; and A projection module configured to implement the projection method according to any one of claims 1 to 5.

8. A computer program product that can be directly loaded into a computer memory, the computer program product comprising software code portions for executing the steps of the projection method according to any one of claims 1 to 5 when the program is executed on a computer.

Images