Portable device for displaying graphical information on a remote object

Abstract

A portable device (10) for displaying graphical information (12) on a remote object (14) comprises a projector (18), in particular a video projector, for projecting the graphical information (12), a camera (20) for capturing the position and / or location of the object (14), a semitransparent mirror (22) having a first surface and an opposing second surface, and a controller (28) for controlling the projector (18) and the camera (20) and for evaluating the images captured by the camera (20). The device (10) furthermore comprises a portable housing (16) in which at least the projector (18), the camera (20) and the semitransparent mirror (22) are accommodated. The projector (18), the camera (20) and the semitransparent mirror (22) are arranged relative to one another such that the first surface of the semitransparent mirror (22) is located in the beam path of the projector (18) and the second surface of the semitransparent mirror (22) is located in the viewing angle of the camera (20). In a coaxial region of the device (10), either the optical axis (24) of the beam path, deflected through reflection by the semitransparent mirror (22), of the projector (18) is coaxial with the optical axis (26) of the light entering the camera (20) that is transmitted by the semitransparent mirror (22), or the optical axis (24) of the beam path, transmitted by the semitransparent mirror (22), of the projector (18) is coaxial with the optical axis (26) of the light entering the camera (20) that is deflected through reflection by the semitransparent mirror (22).

Description

[0001] Portable device for displaying graphic information on a remote object

[0002] The invention relates to a portable device for displaying graphic information on a remote object.

[0003] WO 2012 / 136345 A2 discloses a system for the visual representation of information on real objects, comprising a projection unit for graphically or pictorially transferring information onto an object. The system comprises a dynamic tracking device with 3D sensors for determining and tracking the pose (position and orientation) of the object, and a control device for the projection unit, which adapts the transmission of information to the current pose of the object determined by the tracking device and displays it in the correct position (or perspective). Such a system can increase the efficiency of manual work steps in manufacturing, assembly, and maintenance while simultaneously improving work quality.By precisely transferring information, such as the digital planning status (CAD model), directly to a workpiece, the time-consuming and error-prone transfer of construction plans using templates and other measuring instruments is eliminated. A visual comparison of target and actual values ​​can be performed intuitively at any time. Furthermore, work instructions, such as step-by-step instructions, can be provided directly on the work object or in the user's field of vision, i.e., exactly where they are actually needed.

[0004] The object of the invention is to provide a more compact, robust and cost-effective projection assistance system that is flexible and versatile in its use.

[0005] This object is achieved by a portable device having the features of claim 1. Advantageous and expedient embodiments of the device according to the invention are specified in the subclaims. The portable device according to the invention for displaying graphic information on a remote object comprises a projector, in particular a video projector, for projecting the graphic information, a camera for detecting the position and / or orientation of the object, a semi-transparent mirror having a first surface and an opposite second surface, and a control device for controlling the projector and the camera and for evaluating the images captured by the camera. The device further comprises a portable housing in which at least the projector, the camera, and the semi-transparent mirror are accommodated.The projector, the camera, and the semitransparent mirror are arranged relative to one another such that the first surface of the semitransparent mirror lies in the beam path of the projector and the second surface of the semitransparent mirror lies in the angle of view of the camera. In a coaxial region of the device, either the optical axis of the projector's beam path deflected by reflection from the semitransparent mirror is coaxial with the optical axis of the incident light from the camera transmitted by the semitransparent mirror, or the optical axis of the projector's beam path transmitted by the semitransparent mirror is coaxial with the optical axis of the incident light from the camera deflected by reflection from the semitransparent mirror.

[0006] For the purposes of the invention, a "portable device" is understood to mean a device in which at least the projector, the camera, and the semi-transparent mirror are housed in a common housing that can generally be carried by a single user and placed in a suitable location, possibly even on a tripod. Of course, a fixed installation is also possible. The device can, for example, be integrated into a production environment, such as on a hall ceiling or a truss.

[0007] In the context of the device according to the invention, an "optical axis" is essentially equivalent to the principal axis of the respective optical system (projector or camera), whereby minor deviations can be covered or corrected algorithmically by a mathematical model, more precisely, the intrinsic calibration. The principal axis (central axis) of a projection cone or light incidence cone deflected by one or more mirrors is also referred to as the optical axis of the corresponding beam path.

[0008] A "semi-transparent" mirror can have different transmission / reflection properties. The ratio of transmission to reflection in the relevant wavelength range can be, for example, 50:50, 60:40, 70:30, 40:60, etc., although of course, neither of the two components can be negligible. It depends on the specific application of the device whether a configuration deviating from the 50:50 ratio is more suitable, for example, to enable better measurements or to direct more light onto the object surface.

[0009] The device's operation also requires the control unit, which can consist of one or more components. As explained in more detail later, key components of the control unit can also be housed within the device's housing or provided separately outside the housing. In the latter case, the external components must be connected (wired or wirelessly) to the projector, camera, and any other electronic components within the device's housing.

[0010] The invention is based on the discovery that beam splitting with a semi-transparent mirror allows for the realization of a particularly compact, mobile projection system with sufficient performance for many applications, simulating a "sequential arrangement" of projector and camera. This means that the projector and camera have an identical perspective, thus avoiding parallax errors and shadowing.

[0011] The device according to the invention opens up a wide range of application areas, such as in the trades and construction sectors, for example, for the correct positioning of electrical / plumbing installations, or in the industrial sector, particularly for assembly support. Thanks to its compact design, the device according to the invention is portable and, thanks to the fixed and protected arrangement of the components in a common housing, also very robust, making it ideally suited for the aforementioned applications. However, certain industrial applications, such as the installation of brackets in confined aircraft fuselage barrels, can only be effectively implemented with such a compact device.

[0012] According to an initial concept, essential components of the control device are also housed in the housing. In this case, the device is particularly mobile, since, apart from the power supply (unless battery operation is planned), only a simple control device or a laptop or tablet, for example, is required for operation. Otherwise, all components required for the device's function are contained in the portable housing, and the device is immediately ready for use.

[0013] According to a second concept, essential components of the control device—in particular, those responsible for the control logic—are deliberately not housed in the housing. The other components of the device housed in the housing can then be arranged even more compactly, allowing this part of the device to be designed even smaller and lighter. This concept also has the advantage that the control device can be designed more flexibly and more easily modified. In this case, before operating the device, the components to be controlled or evaluated must be connected to the components of the control device located outside the housing.

[0014] In a particularly advantageous development of the device according to the invention, a scanning system with a mirror arrangement movable by a drive is provided in the coaxial region of the device, by means of which the coaxial optical axes of the projector and the camera can be deflected together in different spatial directions, wherein the drive is controlled by the control device. By appropriately controlling the mirror device, the projection field of the projector and the field of view of the camera can be considerably expanded. It should, of course, be noted that illumination is not possible across the entire accessible area simultaneously. In many applications, however, sequential illumination is entirely sufficient. According to a first variant of the scanning system, the mirror arrangement has two rotating mirrors and the drive has two galvanometer drives. This scanning technology is proven and enables very fast deflections.

[0015] According to a second variant of the scanning system, the mirror arrangement comprises only one scanning mirror that can be rotated around at least two axes. Such scanning mirrors are already commercially available for many common large-scale projectors, and the concept is also transferable to smaller projectors.

[0016] In a further particularly advantageous development of the invention, a lens unit is provided in the coaxial area of ​​the device, by means of which the beam path of the projector and the light incidence of the camera can be automatically focused and / or automatically zoomed (automatically changing the focal lengths of the camera and projector). By adjusting the lens unit accordingly, the working distance (distance between the device and the object onto which the projection is made) is automatically selected or automatically changed as required.

[0017] Ideally, the lens unit provided in the coaxial area of ​​the device is sufficient for the intended application, so the projector and camera do not need their own lenses. The device can then be designed in a simpler and more compact manner, for example, by positioning the projector's DLP chip and the camera's CCD / CMOS sensor on their respective axes in the beam path at a consistent distance from the lens.

[0018] The projector of the device is preferably a video projector. Especially in conjunction with a movable mirror arrangement, a very compact, so-called "pico" video projector can be used. It has a comparatively low light output and a longer focal length, thus providing a smaller projection field at a given working distance. Such pico video projectors are not only inexpensive but also very energy-efficient and, due to their small dimensions and low heat generation, are easy to protect.

[0019] A laser projector, particularly one with a point laser source, can also be used as the projector. In an alternative, cost-effective embodiment, a compact light source is used as the projector, which projects a simple geometric shape, such as a circle or a cross. The light source can be an LED pattern projector, which can generate various shapes using interchangeable pattern masks, or a laser diode, which can generate different patterns using a diffractive optical element. Particularly in conjunction with a scanning system, cost advantages can be achieved compared to a laser projector, as a slower deflection mechanism is sufficient. With a point laser source, the impression of a stationary contour does not have to be created, but rather the pattern already projected by the light source only needs to be directed to the desired position.A further advantage of the lower acceleration values ​​is the reduced wear, thus allowing for a longer service life. Although this design offers little flexibility in terms of the achievable projection content (fixed configuration or a few patterns statically configurable via pattern masks), it is sufficient for a wide range of worker guidance applications.

[0020] The device's camera is typically a conventional 2D camera. In a more sophisticated embodiment, a depth camera (3D camera or time-of-flight (ToF) camera) can be used instead. With such a camera, a point cloud is continuously generated on the targeted section of the component with a high update rate. The ToF camera typically operates in the infrared (IR) range. The light is emitted and evaluated for depth information by means of time-of-flight measurement or interferometrically by correlating the outgoing light wave with the reflected light wave on the detector. The measurement is performed on a sensor with many measuring points (pixels). If the sensor has enough pixels, it also generates a 2D image with intensity values ​​in the IR range. This means that 2D and 3D information is available for each pixel.

[0021] The 3D image can be used for evaluation as an alternative to or in addition to the 2D camera image. “Evaluation” refers to “registration” or “tracking,” as well as “detection / verification of the correct assembly of attachments.” Depending on the type of component, 3D data may be better suited for these applications. This is particularly the case if the component does not have clear and unambiguous corners, edges, holes, or the like that could be clearly segmented in the 2D camera image, but rather has a homogeneous, continuously curved surface. On such surfaces, 3D point clouds can simply be “snapped” for “referencing / tracking,” or directly compared with the 3D model for “detection / verification.” It should be noted that “snapping” requires the presence of elements in all three spatial directions; therefore, it does not work, for example, with a 3D point cloud.on a planar table surface (because the point cloud can be moved freely on it), but very well on convex or concave curved components. Since ToF cameras have significantly lower resolutions (typically 640x480 pixels) than conventional 2D cameras, a design with deflection of the field of view via a movable mirror arrangement with one or more mirrors is particularly advantageous, as this still allows a high resolution to be achieved on the targeted image area on the component. In principle, this resolution-related advantage is similar to the use of a pico video projector and, in particular, allows for greater precision.

[0022] To make the device according to the invention even more compact and cost-effective, a simple laser distance meter can be used instead of a ToF camera, which determines the distance to the object onto which the projection is made in one direction. The distance measurement can be based on laser time-of-flight measurement or interferometry. In simple terms, such a laser distance meter is functionally equivalent to a ToF camera with only one pixel. With this measuring principle, only one depth measurement can be taken at a time; however, by deflecting it using a scanning system, different spatial directions and thus points in the target environment can be approached sequentially (instead of in parallel for many pixels of the ToF camera).Such an embodiment is particularly advantageous when only a few points are required, for example for referencing the distant object (workpiece) or for a simple check of the presence of attachments (e.g., is a plug already in the hole?). In this case, the measurement points can be determined in a sufficiently short time even with a relatively slow deflection unit (independent of the number of mirrors in the scanning system). The projector of the device according to the invention can also comprise several different projector units, namely a video projector unit and / or a pico-video projector unit and / or a laser projector unit and / or an LED pattern projector unit and / or a laser diode unit.With such a multifunctional projector, the projection method most suitable for the respective application can be used either individually or a combination of different projector units can be used simultaneously, which also opens up new application possibilities that would not be possible with a single projector unit.

[0023] A particularly advantageous combination is achieved with a simple LED pattern projector unit and a laser projector unit with a point laser source featuring a diffractive optical element. A device with these two projector units represents an ultra-compact and extremely cost-effective solution for worker guidance in somewhat less dynamic contexts. The LED pattern projector unit and the depth sensor are either coupled into a beam path via the semi-transparent mirror (as with the other variants already described), whereby the point laser source can be used simultaneously for a time-of-flight measurement or an interferometric measurement with the help of the control device, thus taking over this functional part of a laser distance meter.In the case of interferometric measurement, the laser light is also guided onto separate optical paths by means of the semi-transparent mirror acting as a beam splitter - to fulfill the different functions - and is then reflected at the respective path end by additional mirrors and finally recombined.

[0024] It is also possible to incorporate several, particularly different, cameras, such as a 2D and a 3D camera, or at least one camera, particularly a 2D camera, together with a laser rangefinder in the device. This allows the device's functional spectrum to be significantly expanded.

[0025] According to a particular embodiment, the semitransparent mirror is provided with a polarization filter. Further features and advantages of the invention will become apparent from the following description and the accompanying drawings, to which reference is made. In the drawings:

[0026] - Figure 1 is a schematic representation of a device according to the invention in a first embodiment;

[0027] - Figure 2 is a schematic representation of a device according to the invention in a second embodiment;

[0028] - Figure 3 is a schematic representation of a device according to the invention in a third embodiment; and

[0029] - Figure 4 is a schematic representation of a device according to the invention in a fourth embodiment.

[0030] Figure 1 shows a simple first embodiment of a portable device 10 for displaying graphic information 12 on a remote object 14. This basic structure can be supplemented by additional components, which will be discussed in more detail below in the description of further, particularly advantageous embodiments with reference to Figures 2 to 4.

[0031] A portable housing 16 contains a projector 18, in particular a video projector, for projecting the graphic information 12, and a camera 20 for detecting the position and / or orientation of the object 14. Instead of a video projector, a laser projector can in principle also be used, with priority being given to embodiments using a video projector below. Furthermore, a semi-transparent mirror 22 is provided in the housing 16, which partially reflects and partially transmits both the light emitted by the projector 18 and the light incident on the camera 20.

[0032] The projector 18, the camera 20, and the semi-transparent mirror 22 are arranged relative to one another such that a first surface of the semi-transparent mirror 22 lies in the beam path of the projector 18, and the opposite second surface of the semi-transparent mirror 22 lies in the viewing angle of the camera 20. In the exemplary embodiment shown in Figure 1, the projector 18 and the camera 20 are oriented such that their optical axes 24 and 26, respectively, are perpendicular to one another, and the semi-transparent mirror 22 is arranged and oriented at the intersection point of the two optical axes 24, 26 such that it is inclined by 45° with respect to both optical axes 24, 26.

[0033] Due to the semitransparent mirror 22, the optical axis 24 of the beam path of the projector 18, deflected by reflection, is changed such that it coincides with the optical axis 26 of the portion of the incident light of the camera 20 transmitted by the semitransparent mirror 22. The region of the device 10 in which the optical axes 24, 26 are coaxial is referred to for simplicity as the coaxial region of the device 10.

[0034] In principle, it is also possible to use the device 10 such that the optical axis 24 of the beam path of the projector 18 transmitted by the semitransparent mirror 22 is directed toward the object 14. In this case, the portion of the light incident on the camera 20 reflected by the semitransparent mirror 22 is of interest.

[0035] In both cases, the beam path of the projector 18 is combined with the beam path of the camera 20 via the semi-transparent mirror 22, so that a series arrangement of projector 18 and camera 20 is simulated.

[0036] A control device 28 is provided for controlling the projector 18 and the camera 20, as well as for evaluating the images captured by the camera 20. The essential components of the control device 28 can be provided inside the housing 16 or—unlike shown in Figure 1—outside the housing 16.

[0037] Projector 18 is intrinsically calibrated, meaning the principal image point, lens distortion, and focal length of projector 18 are known. Likewise, camera 20 is intrinsically calibrated, meaning the principal image point, lens distortion, and focal length of camera 20 are also known. This intrinsic calibration is known and described, for example, in J. Weng, P. Cohen, M. Herniou: Camera Calibration with Distortion Models and Accuracy Evaluation; IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 14, No. 10, pp. 965-980 (1992). Furthermore, camera 20 is extrinsically calibrated to projector 18, meaning its pose (position / orientation) is known.

[0038] Using the camera 20, the spatial relationship between the object 14 and the projector 18 can therefore be determined (registration). This can also be done continuously (tracking). Methods based on markers or the like 30 (see, for example, WO 2012 / 136345 A2) and markerless methods (see, for example, WO 2022 / 136030 A2) are possible for this purpose.

[0039] By means of this arrangement, graphic information 12 can be projected onto the object 14. This graphic information 12 can be used to display digital content in the sense of augmented reality in the correct position on the object 14.

[0040] The calibration of a 2D camera to a projector is known, for example, from WO 2012 / 136345 A2 and DE 10 2016 105 405 A1. It is essentially based on the observation of multiple projections relative to precisely measured points on a planar or nonplanar test field using the camera. This allows both the intrinsic and extrinsic parameters of a projector-camera system to be determined in a single step. The prerequisite is a sufficient number of observations in a mathematically non-degenerate constellation so that the parameters to be determined are sufficiently determined (ideally overdetermined) and can be determined by iteratively solving the nonlinear functional relationship (bundle block adjustment).

[0041] Of particular importance here is the combination of the projector 18 and the camera 20 via the semi-transparent mirror 22. Since the projector 18 and the camera 20 are optically located in the same beam path, the camera 20 has exactly the same perspective on the object 14 as the projector 18. If the semi-transparent mirror 22 is sufficiently thin and planar, the calibration procedure for the projector-camera system described above remains fundamentally unchanged.

[0042] Although the device 10 is therefore not suitable for performing a depth measurement by triangulating an element (point or similar) projected onto the surface of the object 14 (scanning, see e.g. WO 2015 / 091291 A1), since the necessary baseline (distance between two cameras or between projector and camera) is missing, the object 14 can nevertheless be illuminated using the projector 18 and viewed from exactly the same perspective. This is particularly advantageous when the object has a complex shape and the observable elements on the surface are highly dependent on the viewing angle. The device 10 is therefore particularly suitable for assessing the correct positioning of elements (e.g. attachments) on the surface of the object 14. The elements can be suitably illuminated by projection (structured lighting) so that they are depicted with high contrast in the camera image.Since the perspective of the structured illumination and the observing camera 20 is identical, there are no parallax effects. This means that complex-shaped surfaces, height differences caused by elements mounted on the object surface, etc., do not lead to shadows. In particular, the illuminated surface can be completely imaged. This allows for particularly robust implementation of image processing algorithms.

[0043] The device 10 therefore makes it possible (i) to save on additional cameras and lighting by allowing different locations to be approached from a single perspective and suitably illuminated by the projector 18, e.g., for robust image processing to detect / check the correct assembly of add-on parts), and (ii) to make production or assembly more flexible by allowing previously unknown positions to be easily controlled digitally (compared to the use of conventional templates), e.g., to display assembly instructions or positioning information (crosshairs, contours, etc.) for the exact positioning of add-on parts, or to display information to identify elements on the component (e.g., one hole among many into which a clip, screw, or similar is to be inserted). While conventional templates have to be procured and kept in order to be able to produce a specific product variant, orTo implement a specific plan, any product variant can be produced by digitally controlling a "digital template" using device 10, provided the necessary 3D data is available. The generation of the projection content can also be fully automated, for example, by automatically creating and displaying the 3D content based on the component identification number and the relevant parts list. This eliminates any manual setup process in a production line.

[0044] It should be noted that, for the sake of simplicity, in Figure 1 (and also in the subsequent figures), the projection cone of projector 18 (solid lines) and the light incidence cone of camera 20 (dashed lines) are identical in the coaxial area and outside of device 10. In this case, the focal lengths of projector 18 and camera 20 are identical. However, this is not mandatory. In fact, different focal lengths can be advantageous for certain applications.

[0045] Figures 2 and 3 show extended embodiments of the portable device 10 in which the basic structure described above is combined with a (symbolically represented) scanning system 31.

[0046] In the variant shown in Figure 2, two rotating mirrors 32, 34 are arranged in the coaxial area of ​​the device 10, which are connected by two

[0047] Galvanometer drives 36, 38 (galvos for short) can be deflected. More specifically, the first rotating mirror 32 can be rotated about a first axis, and the second rotating mirror 34 can be rotated about a second axis that is at least approximately perpendicular to the first axis. The galvos 36, 38 are controlled by the control device 28.

[0048] By means of the rotating mirrors 32, 34, the common beam path of the projector 18 and the camera 20 can be deflected in two mutually perpendicular directions. Thus, the area in the environment reachable by the beam path can be significantly expanded by widening the nominal projection field of the projector 18 or the nominal viewing cone of the camera 20 by deflecting the rotating mirrors 32, 34. Common galvo-scan systems achieve deflection angles of + / -20°. 0 , which allows an optical range of 80° (angle of incidence = angle of reflection, thus doubling the deflection).

[0049] Specifically, the device 10 can therefore combine a very compact pico video projector 18 with low light output and a larger focal length (e.g., 5°, equivalent to an image width of approximately 17 cm at a projection distance of 2 m), resulting in a smaller projection field for a given working distance. Pico video projectors with low light output are not only very inexpensive, but also compact in design and, due to their low energy consumption, particularly economical. Due to the lower waste heat, the device 10 is also much easier to cool. In summary, such an arrangement creates a very compact, lightweight unit that is easy to protect against environmental influences such as dust and moisture, and thus inexpensive to manufacture, yet capable of illuminating large areas.

[0050] Illumination cannot be achieved simultaneously across the entire area, which makes it unusable for some applications, such as when larger structures (e.g., a CAD layout for the precise arrangement of cut pieces on a table) need to be displayed at once. However, in many other applications, sequential illumination is sufficient (e.g., to display specific steps of a work instruction one after the other).

[0051] The unit essentially acts like a flashlight, automatically directing the viewer to where they want to look. The same light output per area is available for each illuminated area (e.g., 500 ANSI lumens per 0.25 m 2 = 2000 ANSI lumens per m 2 ), as would be the case when illuminating a larger area with a much larger, more powerful, more expensive device (e.g. 4000 ANSI lumens per 2 m 2 = 2000 ANSI lumens per m 2). Here, it is important to consider that for a high-contrast display, the guideline value is at least three to five times the illuminance on the object surface, measured in lux, in ANSI lumens per m 2 must be expended. For typical workplace environments, the illuminance from artificial light is at least 500 lux, and in some workplaces, significantly more. In work environments where natural sunlight can penetrate, the illuminance can be 2000 lux or more. It is therefore easy to see that there are strict limits to the complete, high-contrast illumination of object surfaces using video projectors in practice (e.g., for a table measuring 2 m x 2 m = 4 m 2 Even with moderate illumination of only 500 lux, at least 4 x 500 lumens / m 2x 4 = 8000 ANSI lumens, which already requires a large projector. This does not even take into account the fact that the common 4:3, 16:9, or 16:10 aspect ratios of commercially available projectors do not necessarily match the object surface, which may result in significantly lower net light output.

[0052] A further advantage over the first embodiment is that the device 10 with the scanning system 31 consumes very little power due to the low light output of the pico video projector 18. This not only benefits the environment but also makes the device 10 much easier to cool due to less waste heat. Conventional video projectors are typically air-cooled, with no decoupling between the environment and the housing interior. Such devices are therefore typically only classified with protection class IP20 (protected against solid foreign bodies with a diameter > 12.5 mm; protected against access with a finger; no protection against water). In particular, environmental influences such as metallic dust and aerosols pose a problem for the longevity of the devices. This means that conventional video projectors cannot be used in dirty environments such as welding shops, construction sites, etc.The device 10, on the other hand, can be designed to be compact, cost-effective, and yet dual-circuit due to its low heat generation, keeping dirt away from the interior of the housing and thus from critical components. This particularly advantageous embodiment is therefore suitable for use as a worker assistance system in a wide variety of environments.

[0053] Another advantage compared to the first embodiment is that, for a given resolution of the projector 18 of the device 10, a higher number of pixels per area (DPI) is available on the object surface due to the longer focal length. This allows for thinner, more delicate, and thus more precise projections to be displayed.

[0054] In a particularly advantageous variant of the device 10, both the projector 18 and the camera 20 are designed with a high focal length. This reduces the field of view available for simultaneous observations of the object space by the camera 20. At the same time, however, the usable resolution on the object surface increases. This allows even finer structures to be resolved. The quality of the data available for image processing algorithms is thereby significantly increased without shrinking the overall (sequentially) observable areas. This offers considerable advantages for all applications based on observation by a camera. This includes not only the precise referencing of the projector 18 relative to the object 14 (or tracking), but also the determination of properties of the object 14, such as determining the correct and precise assembly of an attachment to a workpiece.The higher available resolution can significantly increase the informative value of such image processing methods.

[0055] As already indicated, a laser projector—also based on a galvo scanner—can be provided as projector 18 instead of a video projector. In such an embodiment, a point laser source is deflected by two mirrors in two mutually perpendicular directions, creating the impression of a stationary laser contour for the observer. Such galvo-assisted laser projectors are frequently used in industry, for example, to transfer precise positioning or trimming to planar tables or complex-shaped components. The galvos generally have a very high resolution of at least 2 16 Bit, which affects the usable deflection angle of + / -20 0and the resulting field of view extends to 80° x 80°. The horizontal resolution of a 4K video projector, in comparison, is only 4096 or 3840 (depending on the image format). However, if this is distributed over a 5° viewing angle instead of 80°, the device 10 results in an effective resolution of approximately 4000 * (80 / 5) = 64000, which is roughly equivalent to the high resolution of 2 16 Bit of the laser projector.

[0056] For clarity, it should be noted that the resolution of the galvos 36, 38 is not crucial for the resulting resolution of the overall arrangement; they only need to roughly align the field of view of the video projector 18 to the object 14 such that the desired target area (ROI) on the object 14 is completely covered by the usable field of view of the projection. After that, only the resolution of the video projection available there is decisive. Calibration and appropriate control ensure that the appropriate pixel is assigned to the desired target coordinate on the object 14. If the device 10 is to be designed for larger working distances, the focal length of the projector 18 simply needs to be increased, resulting in a smaller viewing angle (e.g., 2.5° instead of 5°). This ensures the same effective resolution even at a greater working distance.This represents a further advantage of the device 10 compared to the aforementioned laser projectors, whose effective resolution is always determined by the resolution of the galvos viewed over the entire image field (e.g. 2. 16 to 80°), whereby the discretization effect becomes increasingly greater with increasing working distance.

[0057] Since the rotating mirrors 32, 34 of the device 10 do not need to move at high frequency to create the impression of a stationary contour, but rather only need to move the image section of the video projector 18 to the desired location, for which slow movements are sufficient, the demands on the galvos 36, 38 are also significantly lower. Significantly less expensive models with low acceleration values ​​can be used. Furthermore, the slow movement and the resulting low forces also result in significantly less wear on the mechanical parts. This allows for a system that is just as precise and also more durable to be manufactured at a lower price.

[0058] Since device 10 cannot illuminate all areas simultaneously, and thus cannot see them, its ability to dynamically track objects is somewhat limited. Referencing or tracking objects typically requires the observation of several markers or other distinctive geometric elements, such as edges, corners, holes, etc., or even distinctive texture elements, such as light / dark transitions, so-called features, generally designated here by reference numeral 30 (at least four on planar surfaces or six on non-planar surfaces, distributed across the object surface). Due to the deflection of rotating mirrors 32, 34, these observations can only be performed sequentially.The latency between two observations depends on the design of the mirror system (size / weight of the rotating mirrors 32, 34 and the performance of the galvos 36, 38) as well as the angle by which the rotating mirrors must be moved between the two observations. In the worst case, the cone of view must be moved from the far left to the far right, or from the very bottom to the very top. With simultaneous movement, "observations" of relevant elements on the object surface cannot be realized simultaneously, but only sequentially. Mathematically speaking, it is therefore not possible to perform four or six observations of reference elements simultaneously, assuming moving objects. With slower movements, however, the four or six observations are sufficiently "simultaneous" for robust referencing. For stationary objects, there are no limitations at all.The technique can also be used for fast-moving objects (such as on an assembly line). The inconsistency in the four or six observations caused by the movement of object 14 then leads to a reduction in the achievable accuracy in the continuous pose determination. For uniform movements (as typically occurs in manufacturing environments, assembly lines, etc.), the inconsistency caused by the temporal offset of the observations can even be completely compensated for by a Kalman filter using a suitable motion model. Assuming uniform or uniformly accelerated movements, the same precision is achieved as for stationary objects.

[0059] In a further expansion stage, the device 10 can be implemented with a so-called depth camera (3D camera or time-of-flight (ToF) camera) instead of a conventional 2D camera. This expands the possibilities, because, as already explained above, arrangements equipped with 2D cameras are not capable of performing triangulation-based depth measurements due to the lack of a baseline. This limitation is eliminated by using a ToF-based depth camera. The measuring principle used here is a time-of-flight measurement of the light emitted by the ToF camera, reflected by the object surface, and finally received again by the ToF camera. The depth results from the time difference between the emitted and received light; the baseline plays no role in this measuring principle. This eliminates the above-mentioned disadvantage of arranging projector 18 and camera 20 in the same beam path.ToF cameras typically work with invisible infrared illumination. The beam path, in particular the semi-transparent mirror 22, must be designed so that infrared radiation is partially transmitted in both directions, as is visible light. If the device 10 provides for a deflection of the beam path of at least the projector 18 or also the camera 20, the calibration of the optical components becomes more complicated. It is no longer sufficient to know the intrinsic properties of the projector 18 and camera 20, as well as the extrinsic relationship between these two components, taking into account the semi-transparent mirror 22. In addition, there are the imaging properties of the two rotating mirrors 32, 34. This can be described mathematically using models that, for example,are described in Eisert, Peter; Polthier, Konrad; Hornegger, Joachim: A mathematical model and calibration procedure for galvanometric laser scanning systems; Vision, Modeling, and Visualization. 2011. pp. 207-214. These models are also used to deflect point laser sources in laser projectors. However, unlike the point laser source, the video projector 18 creates a flat image. With a naive implementation, a point (such as the image center or principal point) can be regarded as analogous to the point laser source. For pixels with increasing distance from this one point, this would result in an increasing error in the mathematical model for a given deflection of the rotating mirrors 32, 34. In order to use a precise mathematical model for all pixels, the calibration must therefore take into account the entire projection area for every possible mirror deflection in order to achieve the highest precision.This is achieved, among other things, by increasing the number of measurements. Just as the deflection of the point laser sources of different laser projectors, each with a characteristic angle of incidence of the point laser on the first and second rotating mirrors, can be calibrated device by device, this can also be done within the device 10 with the video projector 18, whereby the different angles of incidence of the point light source on the first and second rotating mirrors 32, 34 are present, so to speak, in the form of different pixels in a single device. In practice, calibration does not have to be performed separately for each pixel. It is sufficient to calibrate for a few pixels distributed across the image and to interpolate the optical path for intermediate pixels.

[0060] Figure 3 shows a further embodiment of the device 10, in which the structure shown in Figure 1 is combined with a scanning system 31. Here, the scanning system 31 is formed by a special scanning mirror 40 with a drive in the form of a deflection mechanism 42 adjustable by electric stepper motors. Thus, compared to the previously described embodiment, the number of mirrors in the scanning system 31 is reduced from two to one. Scanning systems with only one deflectable scanning mirror are available for larger projectors, for example, from the Dynamic Projection Institute under the designation "Mirror Head."

[0061] With such a scanning mirror 40, the projection of projector 18 can be deflected considerably further than the lens shift functionality typically built into commercially available projectors. The scanning mirror 40 can be deflected around two mutually perpendicular axes by the deflection mechanism 42 controlled by the control device 28. Due to the more complex deflection mechanism 42 and its correspondingly greater mass compared to the galvos 36, 38, the frequency of possible mirror movements decreases. However, for some (static or slow-moving applications), this does not pose a problem.

[0062] The available scanning mirror systems are not calibrated, but the image can be reproducibly directed to different locations in the room. Since the scanning mirror 40 and typically the projection surface are planar, the perspectively distorted image can be rectified using simple homography. This is achieved without in-depth mathematical 3D calibration by manually adjusting the four image corners in their respective horizontal and vertical deflections. It is sufficient to be able to reproducibly align the image to the same surface at any time "at the push of a button."

[0063] In the context of this invention, such a scanning system 31 with a scanning mirror 40 represents a special case of the previously described scanning system 31 with two rotating mirrors 32, 34, in which the mirror spacing is 0. This eliminates one of the parameters that must be calibrated during the calibration of two-mirror systems. This allows for significantly larger aperture angles of up to 180°, which is advantageous in the example of the aircraft fuselage barrel (see above). The entire half-shell can be reached laterally from a single position.

[0064] In another particularly advantageous variant, the scanning mirror 40 is not operated mechanically via moving parts, but is designed as a so-called MEMS (microelectromechanical system). The scanning mirror 40 is then arranged (in the zero position) at an inclination of 45° in the beam path of the projector 18 and the camera 20. The disadvantage is that the inclined mounting of the scanning mirror 40 results in a strong perspective, with decreasing image resolution as the deflection increases. The advantage is that there are no mechanical parts and therefore no wearing parts. The system is more compact overall. In conjunction with the arrangement of projector 18 and camera 20 in one beam path described above, the invention can thus also be implemented using the MEMS.

[0065] Finally, Figure 4 shows another particularly advantageous embodiment in which the combined beam path of projector 18 and camera 20 can be focused and zoomed via a lens unit 44. This allows for variable working distances. If the lens unit 44 is motorized, different working distances can be automatically imaged in a single process. If necessary, using such a lens unit 44 can completely eliminate the need for lenses on the projector 18 and camera 20, allowing the device to be designed even simpler and more compact.

[0066] Particularly for simpler applications, instead of a video projector or a laser projector unit with a point laser source, which must be quickly deflected using a galvo scanner in order to give the viewer the impression of a stationary projection on a component, a cost-effective light source can be used as projector 18. This light source projects a simple geometric shape, such as a circle or a cross, by means of which an element or a location on the component can be clearly identified. Such a light source can in particular be an LED pattern projector, which can generate precise, intense and uniform lines, grids, crosses and point clouds using easily exchangeable pattern masks, whereby the various masks can be integrated via plug & play. Alternatively, a laser diode can be used as the light source, which can generate various patterns using a diffractive optical element.

[0067] The advantage of such a design is that, instead of using two rapidly moving mirrors using galvo scanners, a slower mechanism for deflecting the light beam is sufficient. This is because the impression of a stationary contour is not required; instead, the pattern already emerging from the light source needs to be aligned to the desired position on the component. Such a slow mechanism can be achieved using one or two mirrors (see explanation in the current patent application) and is potentially more cost-effective. Due to lower acceleration values, it also results in less wear, thus allowing for longer service lives.

[0068] The semitransparent mirror 22 can be of various types. Some preferred variants are explained in more detail below.

[0069] The ratio of transmitted to reflected light (reflection / transmission) is typically about 50:50, although a different ratio can be chosen depending on the application.

[0070] The semi-transparent mirror 22 can be implemented with or without a polarization filter. A polarization filter can minimize reflections on reflective surfaces, resulting in more robust measurements and higher data quality.

[0071] The wavelength range in which the semitransparent mirror 22 is transmissive should be matched to the projector 18 and the camera 20 used. By default, all typical wavelength ranges could be transmitted.

[0072] Depending on the design variant, the following advantageous combinations can be defined:

[0073] - 2D camera + video projector: semi-transparent mirror 22 is permeable to visible light.

[0074] - 2D camera + monochromatic LED projector or laser diode: semi-transparent mirror 22 is “narrowband”, permeable only in the wavelength range of the light source.

[0075] - 3D ToF camera + video projector: semi-transparent mirror 22 transmits visible and infrared light. - 3D ToF camera + monochromatic LED projector or laser diode: semi-transparent mirror 22 is "narrowband," transmitting only the wavelength range of the light source and infrared light.

[0076] - 2D camera + monochromatic video projector: semi-transparent mirror 22 is "narrow-band", permeable only in the wavelength range of the light source (see also blue light scanner from photogrammetric measurement technology, particularly high signal-to-noise ratio for measurement tasks).

[0077] - 3D camera + monochromatic video projector: semi-transparent mirror 22 is "narrow-band", permeable only in the wavelength range of the light source (see also blue light scanner from photogrammetric measurement technology, particularly high signal-to-noise ratio for measurement tasks) and for infrared light.

[0078] - 2D camera + laser diode with dual function “projection” and “for laser distance measurement”: semi-transparent mirror 22 is “narrowband”, permeable only in the wavelength range of the light source.

[0079] In general, the smaller the wavelength range of light that can pass through the semitransparent mirror 22, the better the signal-to-noise ratio. However, the display quality suffers if multicolor display is not possible.

[0080] List of reference symbols

[0081] 10 Device

[0082] 12 graphic information

[0083] 14 objects

[0084] 16 housings

[0085] 18 projector

[0086] 20 Camera

[0087] 22 semi-transparent mirror

[0088] 24 optical axis of the projector

[0089] 26 optical axis of the camera

[0090] 28 Control device

[0091] 30 markers

[0092] 31 Scan system

[0093] 32 first rotating mirror

[0094] 34 second rotating mirror

[0095] 36 first galvanometer drive

[0096] 38 second galvanometer drive

[0097] 40 scanning mirrors

[0098] 42 Deflection mechanism

[0099] 44 Lens unit

Claims

Patent claims 1. A portable device (10) for displaying graphic information (12) on a remote object (14), comprising a projector (18), in particular a video projector, for projecting the graphic information (12), a camera (20) for detecting the position and / or orientation of the object (14), a semitransparent mirror (22) having a first surface and an opposite second surface, a control device (28) for controlling the projector (18) and the camera (20) and for evaluating the images captured by the camera (20), and a portable housing (16) in which at least the projector (18), the camera (20), and the semitransparent mirror (22) are accommodated, wherein the projector (18), the camera (20), and the semitransparent mirror (22) are arranged relative to one another in such a way thatthat the first surface of the semi-transparent mirror (22) lies in the beam path of the projector (18) and the second surface of the semi-transparent mirror (22) lies in the viewing angle of the camera (20), and that in a coaxial region of the device (10), either the optical axis (24) of the beam path of the projector (18) deflected by reflection from the semi-transparent mirror (22) is coaxial with the optical axis (26) of the incident light of the camera (20) transmitted by the semi-transparent mirror (22), or the optical axis (24) of the beam path of the projector (18) transmitted by the semi-transparent mirror (22) is coaxial with the optical axis (26) of the incident light of the camera (20) deflected by reflection from the semi-transparent mirror (22).

2. Device (10) according to claim 1, characterized in that essential components of the control device (28) are not accommodated in the housing (16).

3. Device (10) according to claim 1 or 2, characterized in that in the coaxial region of the device (10) a mirror arrangement (32, 34; 40) is provided which is movable by a drive (36, 38; 42), by means of which the coaxial optical axes (24, 26) of the projector (18) and the camera (20) can be deflected together in different spatial directions, wherein the drive (36, 38; 42) is controlled by the control device (28).

4. Device (10) according to claim 3, characterized in that the mirror arrangement has two rotating mirrors (32, 34) and the drive has two galvanometer drives (36, 38).

5. Device (10) according to claim 3, characterized in that the mirror arrangement has a scanning mirror (40) rotatable about at least two axes.

6. Device (10) according to claim 5, characterized in that the scanning mirror (40) is rotatably mounted in a holder which is attached to the projector (18).

7. Device (10) according to one of claims 3 to 6, characterized in that a lens unit (44) is provided in the coaxial region of the device (10), by means of which the beam path of the projector (18) and the light incidence of the camera (20) can be automatically focused and / or automatically zoomed at the same time.

8. Device (10) according to claim 7, characterized in that the projector (18) and the camera (20) do not have their own lenses, apart from the lens unit (44) in the coaxial region of the device (10).

9. Device according to one of the preceding claims, characterized in that the projector (18) comprises a pico video projector unit.

10. Device according to one of the preceding claims, characterized in that the projector (18) comprises a laser projector unit, in particular with a point laser source.

11. Device according to one of the preceding claims, characterized in that the projector (18) comprises a light source which projects a simple geometric shape, the light source being an LED A pattern projector unit that can produce different shapes using interchangeable pattern masks, or a laser diode unit that can produce different patterns using a diffractive optical element.

12. Device according to one of the preceding claims, characterized in that the projector (18) comprises at least two of the following projector units: pico video projector unit; laser projector unit; LED pattern projector unit; laser diode unit.

13. Device according to one of the preceding claims, characterized in that the camera (20) comprises a depth camera.

14. Device according to one of the preceding claims, characterized in that a laser distance meter is provided in addition to or instead of the camera (20).

15. Device according to one of the preceding claims, characterized in that in addition to the camera (20) at least one further camera or a laser distance meter is provided.

16. Device according to claim 10 and claim 14 or 15, characterized in that the point laser source of the projector (18) is also a component of the laser distance meter and is used to measure the distance of the object (14).

17. Device according to one of the preceding claims, characterized in that the semi-transparent mirror (22) is provided with a polarization filter.

Images