Localization system and localization method with markers having holographic optical elements

Abstract

The invention discloses a concept for improved position and orientation determination of an object marked with multiple optoelectronically detectable markers, in which holographic optical elements are incorporated into two-dimensional, planar markers, at which ambient light is reflected back with sufficiently high intensity even at high scattering angles relative to the surface normal of the two-dimensional planar markers. For this purpose, it is proposed that, in an optoelectronic sensor (32), holographic optical elements (HOE;21, 22, 23) and to generate a temporal sequence of two-dimensional reflection images, and to detect in an image processing device (34) on the basis of the spectrally resolved local intensity distribution of the sequence of two-dimensional reflection images detected by the optoelectronic sensor (32) relative differences between the reflection behavior of a first of the optoelectronically detectable markers (20) and the reflection behavior of a second of the optoelectronically detectable markers (20) or between the reflection behavior of a first of the optoelectronically detectable markers (20) in two temporally successive reflection images and to determine a position and orientation of the object (100; 200; 300; 400; 500; 600) relative to the optoelectronic detection device (30) by evaluating the detected differences.

Description

Technical area

[0001] The present invention relates to a localization system and a localization method based on angle-dependent reflection measurements of holographic optical elements as part of markers to be located. State of the art

[0002] Various positioning methods can be used to determine the position and / or orientation of objects. Optical object recognition, for example, often uses high-contrast geometric structures that can be recognized easily, precisely, and reliably. Examples include optoelectronically detectable color-contrasting markings consisting of differently sized geometric symbols or shapes, such as lines or polygons, with gaps in between, so that the luminance difference at the boundaries between shapes and gaps is as high as possible. Such markings can have one-dimensional coding (e.g., barcodes) or two-dimensional coding (e.g., DataMatrix codes, QR codes, or Aztec codes), each with a different useful information density.The data encoded in a marker can be machine-readable and further processed electronically using optical readers such as cameras or camera scanners.

[0003] Markers can be equipped with holographic optical elements (HOEs) to increase information density. Examples of such markers are disclosed in US 2021 / 0248338 A1, PT 99405 A, CZ 2004-1136 A3, and US 10 885 413 B1. Disclosure of the invention

[0004] The present invention provides a localization system and a localization method, each having the features of the independent claims. Further advantageous embodiments are the subject of the dependent claims.

[0005] A localization system for determining the position and / or orientation of an object marked with multiple optoelectronically detectable markers comprises an optoelectronic detection device. This device has an optoelectronic sensor and an image processing device coupled to the optoelectronic sensor. The optoelectronic sensor is designed to detect light emanating from holographic optical elements applied to the markers in a time-resolved manner and to generate a time-resolved sequence of two-dimensional reflection images, with the light being reflected back to the optoelectronic sensor by the holographic optical element.The image processing device is designed to detect, on the basis of the spectrally resolved local intensity distribution of the sequence of two-dimensional reflection images detected by the optoelectronic sensor, relative differences between the reflection behavior of a first of the optoelectronically detectable markers and the reflection behavior of a second of the optoelectronically detectable markers, and / or relative differences between the reflection behavior of a first of the optoelectronically detectable markers in two temporally successive reflection images, and preferably to determine a position and / or orientation of the object relative to the optoelectronic detection device by evaluating the detected differences.The relative differences particularly concern relative differences in the relative position and / or the relative orientation between the two detected markers and / or the marker in two temporally successive reflection images.

[0006] A localization method for determining the position and / or orientation of an object marked with multiple optoelectronically detectable markers comprises the steps of: time-resolved detection of light emitted by holographic optical elements applied to the markers by means of an optoelectronic sensor of an optoelectronic detection device, wherein the light is reflected by the holographic optical element onto the optoelectronic sensor; generating a time-resolved sequence of two-dimensional reflection images by the optoelectronic sensor from the detected reflected light;Detecting relative differences between the reflection behavior of a first of the optoelectronically detectable markers and the reflection behavior of a second of the optoelectronically detectable markers, and / or detecting relative differences between the reflection behavior of a first of the optoelectronically detectable markers in two temporally successive reflection images, each based on the spectrally resolved local intensity distribution in the sequence of two-dimensional reflection images detected by the optoelectronic sensor; and preferably determining a position and orientation of the object relative to the optoelectronic detection device by evaluating the detected differences.

[0007] Holographic optical elements (HOEs) are optically active structures whose functional principle is based on holography. In an HOE, the imaging properties of a hologram are utilized, or conventional optics are implemented using a hologram, so that light waves can be deflected, split, focused, and / or expanded. HOEs can also be formed, for example, from extremely thin film foils that can decompose incident light into its spectral colors. However, only the light rays that fall within a specific wavelength range are diffracted. By using HOEs, diffraction maxima can be generated, which arise due to constructive interference of light waves at the HOEs.

[0008] In contrast to conventional optics, HOEs deflect light not through refraction, but through diffraction at the volume grating. HOEs can be manufactured for both transmission and reflection, enabling new designs through a free choice of angles of incidence and reflection, or diffraction angle. The holographic diffraction grating is, for example, exposed into a thin, photosensitive film. Volume diffraction can also give a HOE a characteristic wavelength and angle selectivity or filter function. Depending on the imaging condition, such as a wavelength or angle, only light from defined directions and with defined wavelengths is diffracted by the structure of the HOE. Advantages of the invention

[0009] According to some embodiments of the localization system and the localization method, the optoelectronic detection device may further comprise a memory coupled to the image processing device and storing a plurality of comparison and / or training data for the acquired two-dimensional reflection images in a database. This enables rapid and reliably reproducible localization of the object from known reference data.

[0010] According to various embodiments of the localization system and the localization method, the image processing device can further perform object recognition of structures of a contrast code of the markers recorded in the two-dimensional reflection images captured by the optoelectronic sensor. The combination of contrast codes, such as QR codes, with holographic optical elements enables the implementation of redundant recognition mechanisms, which improves robustness against sources of interference, reduces susceptibility to misdeterminations, and increases the accuracy of position and distance determination.

[0011] According to several embodiments of the localization system and the localization method, at least one of the holographic optical elements applied to the markers can have a holographic deflector with high angular selectivity at low elevation angles relative to the surface of the marker. This enables the implementation of a warning function in the event of insufficient detectable effective area of ​​the marker in unfavorable orientations.

[0012] According to some embodiments of the localization system and the localization method, the color depth of the optoelectronic sensor can be at least 8 bits. This advantageously increases the spatial and angular resolution of the position and orientation determination.

[0013] According to some embodiments of the localization system and the localization method, at least one of the holographic optical elements applied to the markers can have an optical function, so that a changing distance of the detection device from the object is indicated by a color change and / or a change in the intensity distribution in the two-dimensional reflection images captured by the optoelectronic sensor. This allows for more precise distance determination, and thus position determination.

[0014] According to some embodiments of the localization system and the localization method, at least one of the holographic optical elements applied to the markers can have an optical function, so that a changing object orientation for dedicated angular ranges is indicated by a color change and / or a change in the intensity distribution in the two-dimensional reflection images captured by the optoelectronic sensor. This allows a more precise determination of the object orientation, and thus a determination of the alignment.

[0015] According to one embodiment of the localization system and the localization method, the holographic optical elements applied to the markers can have a function that replicates a retroreflective function of conventional optics. Such retroreflective functions of conventional optics, i.e., optical functions that reflect incident light back to the illuminating light source, can be found, for example, in cat's eyes or road signs. This can promote clear and stable detection of the markers even at great distances from markers to optoelectronic sensors by using an additional light source, for example, in a time-discrete mode, to illuminate the scene.This advantageously enables a higher detectable intensity and associated increased accuracy, greater possible distances from markers to the detection device, and an improvement in the unique identifiability of the markers when a separate illumination source is used.

[0016] According to some embodiments of the localization system and the localization method, at least two of the holographic optical elements applied to the markers can have duplicated optical functions to improve the reliability of determining the distance of the detection device to the object and / or determining the object orientation for dedicated angular ranges. This advantageously enables particularly robust distance and object orientation determination, since the evaluated back reflections of the at least two holographic optical elements can be mutually validated.

[0017] The above embodiments and developments can be combined with one another as desired, where appropriate. Further embodiments, developments, and implementations of the invention also include combinations of features of the invention not explicitly mentioned above or described below with respect to the exemplary embodiments. In particular, those skilled in the art will also add individual aspects as improvements or additions to the respective basic forms of the invention. Short description of the drawings

[0018] Further features and advantages of the invention are explained below with reference to the figures. Fig. 1: an exemplary illustration of an optoelectronically detectable marker in the form of a QR code; Fig. 2: a schematic illustration of an optoelectronically detectable marker in the form of a QR code with holographic optical elements superimposed on the coding according to one embodiment; Fig. 3: a schematic illustration of the optoelectronic detection of the marker of the Fig. 2 using a localization system according to an embodiment; Fig. 4: an exemplary diagram of a two-dimensionally resolved optoelectronic measurement signal distribution of the holographic optical elements of the marker of the Fig. 2 in a localization system of Fig. 3; Fig. 5: another exemplary diagram of a two-dimensionally resolved optoelectronic measurement signal distribution of the holographic optical elements of the marker of the Fig. 2 in a localization system of Fig. 3; Fig. 6: a flowchart of a localization method for time-resolved location and position determination of an object marked with several optoelectronically detectable markers according to one embodiment; Fig. 7: exemplary structures for redundant holographic optical elements on an optoelectronically detectable marker according to an embodiment; Fig. 8: a schematic illustration of an optoelectronically detectable marker in the form of a QR code with holographic optical elements superimposed on the coding and a holographic deflector according to a further embodiment; Fig. 9: a schematic illustration of an optoelectronically detectable marker in the form of a QR code with holographic optical elements superimposed on the coding according to a further embodiment; Fig. 10: a schematic plot of the reflection intensity maxima along the distance from the optoelectronically detectable marker of the Fig. 9; Fig. 11: a schematic plot of the frequency-dependent reflection intensity along the distance from the optoelectronically detectable marker of the Fig. 9; Fig. 12: a schematically illustrated diagram of a structure marked with a plurality of optoelectronically detectable markers and an associated optoelectronic detection device according to an embodiment; Fig. 13: a schematically illustrated diagram of a land vehicle marked with a plurality of optoelectronically detectable markers and an associated optoelectronic detection device according to an embodiment; Fig. 14: a schematically illustrated diagram of a machine marked with a plurality of optoelectronically detectable markers and an associated optoelectronic detection device according to an embodiment; Fig. 15: a schematically illustrated diagram of a machine marked with at least one optoelectronically detectable marker and an associated optoelectronic detection device according to an embodiment; Fig. 16: a schematically illustrated diagram of a component marked with a plurality of optoelectronically detectable markers and an associated optoelectronic detection device according to an embodiment; and Fig. 17: a schematically illustrated diagram of a pressurizable surface marked with a plurality of optoelectronically detectable markers and an associated optoelectronic detection device according to an embodiment. Description of embodiments

[0019] Fig. Figure 1 shows an exemplary illustration of an optoelectronically detectable marker in the form of a QR code 10. The exemplary QR code 10 is a deliberately constructed pattern of high-contrast geometric structures. The polygons are optoelectronically detectable and color-contrasted with gaps in between, so that the luminance difference at the boundaries between shapes and gaps is as high as possible. Such markings can have one-dimensional coding (e.g., barcodes) or two-dimensional coding (e.g., DataMatrix codes, QR codes, or Aztec codes), each with a different payload density. The data encoded in such a marker can be machine-readable using optical reading devices, such as cameras or camera scanners, and further processed electronically.

[0020] Determining the location and / or orientation of objects marked by such optoelectronically detectable markers can be error-prone and often inaccurate. The accuracy of the location and / or orientation determination often varies with the distance to the marked object and the lighting conditions in the respective environment. Determining the angular orientation, i.e., the spatial alignment to the optical reader, can also be severely impaired. As distances increase, higher-resolution optoelectronic detection devices may be required. Detecting and locating the patterns on such markers requires the use of complex object recognition algorithms that require high computing power.

[0021] The use of markers equipped with holographic optical elements (HOEs) can improve the localization of the objects marked with them. In particular, but not exclusively, position detection can be significantly improved through optimized angular orientation determination. Better angular resolution, potentially better spatial resolution, lower costs due to lower camera resolution, computationally efficient and fast position determination, as well as possible extensions of existing localization solutions can be achieved through the use of optoelectronically detectable markers in the form of one- or two-dimensional, flat contrast codes such as barcodes, DataMatrix codes, QR codes, or Aztec codes, with holographic optical elements (HOEs) superimposed on the code.

[0022] Fig. Figure 2 shows an example of such an optoelectronically detectable marker 20 with a contrast coding, here a QR code 10. Overlaid on the QR code 10 are HOEs 21, 22 and 23, which are spatially applied to the marker 20 at predefined locations along the surface area of ​​the QR code 10. The number, size, shape, type and position of the HOEs 21, 22 and 23 is shown in Fig. 2 is shown only as an example. It is possible to provide more or fewer than three HOEs 21, 22 and 23 with a different size, shape and / or position on the marker 20.

[0023] The optical function of a hologram is created by the diffraction of incident optical wavefronts by a diffraction grating in the volume of the holographic film. The diffraction grating was previously exposed into the volume of the holographic film during an exposure process. The diffraction grating forms according to an interference pattern generated by two optical recording wavefronts as a modulation of the refractive index in the holographic film. The interference pattern is generated by the superposition of two coherent recording waves in the volume of the holographic film. The interference of the two recording wavefronts creates an interference grating in the volume of the holographic film, which is permanently imprinted into the volume of the holographic film by the exposure process. The parameters of the exposure process (wavefront shape, spectral properties, intensity distribution, and duration) determine the properties of the generated diffraction grating.HOEs can be produced, for example, as hologram structures by holographic wavefront printing, as taught in the document DE 10 2017 218 544 A1.

[0024] Fig. Figure 3 shows a schematic illustration of the optoelectronic detection of the marker 20 of the Fig. 2 using a localization system 1. The localization system 1 comprises an optoelectronic detection device 30, such as a video camera, a camera scanner, or a similar device. Ambient light emitted and reflected by the marker 20 and in particular the HOEs 21, 22, and 23 is recorded via suitable detection optics 31 and fed to an optoelectronic sensor 32. The optoelectronic sensor 32 can, for example, be a CCD sensor, a CMOS sensor, an active pixel sensor (APS), a photodiode array, or another suitable electromagnetic radiation detector with spatial resolution.

[0025] The image captured by the optoelectronic sensor 32 is transferred to an image processing device 34, which evaluates the captured image and transmits the evaluation result to an output interface 35 of the capture device 35. The image processing device 34 can be, for example, a microprocessor, a graphics processor, an ASIC, an FPGA, or any other suitable computing device. The output interface 35 can be, for example, a display or another user interface. It may also be possible for the output interface 35 to establish a communication connection to another device, to which the evaluation result of the image processing device 34 is transmitted via the output interface 35.

[0026] The optoelectronic detection device 30 may also include a memory 33, such as a ROM, a RAM, or another type of data storage device, in which, on the one hand, configuration parameters and operating software for operating the detection device 35 and, in particular, the image processing device 34 are stored. On the other hand, comparison and / or training data can be stored in the memory 33 in a database that enables a comparison of the image captured by the optoelectronic sensor 32 with known images or optoelectronic detection signatures of markers in different positions, orientations, and / or locations.

[0027] The optoelectronic detection device 30 serves, on the one hand, for the general identification and localization of an object marked with a marker 20 based on the contrast coding, such as the QR code 10. For this purpose, the image processing device 34 performs object recognition of the structures of the QR code 10 captured in the image captured by the optoelectronic sensor 32 and compares them with known QR codes, for example, by comparing them with known QR codes stored in a database of the memory 33. Subsequently, the spatial orientation can be deduced by determining the size, distance, and mutual orientation of the patterns and structures of the QR code 10.

[0028] Additionally or alternatively, the image captured by the optoelectronic sensor 32 may include back-reflected light from the HOEs 21, 22, and 23. In addition to the geometric position of the structure of the HOEs 21, 22, and 23, the position and orientation of the object relative to the optoelectronic detection device 30 may be encoded by the color of the back-scattered light. Fig. 4 and Fig. 5 show two exemplary diagrams of two-dimensionally resolved optoelectronic measurement signal distributions P1 and P2 of the HOEs 21, 22 and 23 of the marker 20 of the Fig. 2 by an optoelectronic detection device 30 of the localization system 1 of the Fig. 3.

[0029] HOEs are structures written into a photo-optical layer that redirect incident light. Depending on how the HOE's structure is designed, its optical function can be determined, for example, a scattering hologram, a retroreflector, a mirror, a concave mirror, or a lens. HOEs can also exhibit high angular or wavelength selectivity in their function. For example, the components of white light are scattered to different degrees in different spatial directions. Therefore, the orientation of the HOE relative to the illumination source can be encoded by the backscattered wavelength and determined using the image captured by the optoelectronic sensor 32 of an optoelectronic detection device 30.The spectrally resolved spatial intensity distribution P of three back reflections A, B and C of different HOEs 21, 22 and 23 can be bijectively mapped onto the spatial function F of the HOEs 21, 22 and 23 on a marker 20 measured at azimuth α and elevation β relative to the surface of the marker 20. For example, in . Fig. 4 from the spectrally resolved spatial intensity distribution PI [I(A),I(B), I(C)] of the location-dependent intensities and colors on the two-dimensional image of the optoelectronic sensor 32, the spatial function F(α1, β1) can be deduced. Analogously, for example, in Fig. 5 From the spectrally resolved local intensity distribution P2[I(A),I(B), I(C)] of the location-dependent intensities and colors on the two-dimensional image of the optoelectronic sensor 32, the spatial function F(α2, β2) can be inferred.

[0030] An optoelectronic sensor 32 of an optoelectronic detection device 30 can, for example, have a color depth of at least 8 bits. This means that the optoelectronic sensor 32 can distinguish between approximately 16 million color nuances. This allows the optoelectronic detection device 30 to perform very precise angle measurements of both the azimuth angle α and the elevation angle β. Because several differently positioned HOEs 21, 22, and 23 on the marker 20 each generate unique, linearly independent color patterns on the optoelectronic sensor 32, the respective associated angular orientation of the marker 20 can be uniquely encoded. If the orientation of the object or the position of the optoelectronic detection device 30 relative to the object changes and thus also the orientation of the marker 20 relative to the optoelectronic detection device 30, the wavelengths of the light backscattered by the HOEs 21, 22 and 23 shift.This also allows conclusions to be drawn about the change in position.

[0031] The localization system can additionally comprise a light source 50, which can be provided as a separate device or as part of the detection device 30. The light source 50 can, for example, emit white light W in a targeted manner onto the marker 20, so that the position of the illumination and recording optics is known a priori. This makes determining the location and angular orientation of the marker 20 based on the backreflection patterns in the image of the optoelectronic sensor 32 easier and more reliable. Furthermore, the light emitted by the light source 50 can be modulated, so that a lock-in mechanism can be used to unambiguously assign the backscattered light and improve the signal-to-noise ratio.

[0032] HOEs 21, 22, and 23 can have optimized optical functions, such as retroreflectivity with a limited angular range, scattering functions with high angular selectivity, or any combination of these functions. Furthermore, HOEs 21, 22, and 23 can be designed for light outside the visible range, such as in the UV or infrared range.

[0033] Fig. Figure 7 shows, by way of example, that special patterns and / or special color combinations of HOEs can be used, so that the distance determination of the detection device 30 to the marker 20 can be improved via pattern recognition with regard to color, shape, and size. Due to the angular selectivity of holograms, a change in angle or the object orientation in space can be precisely encoded. For example, one and the same optical function can be applied multiple times to the marker surface. The alignment of wavelengths and intensities of the redundant optical functions can implement a control functionality. Alternatively, the optical function of the surfaces can also be selected such that the intensities of the back-diffracted light must differ. The deflection function can, for example, also be designed in the opposite direction, resulting in a point-symmetric intensity distribution of the back-diffracted light.If the position of marker 20 relative to the detection device 30 changes, this distribution must also change point-symmetrically. Areas addressing different wavelengths can be arranged symmetrically or asymmetrically, whereby the color composition of the reflected light also changes symmetrically or asymmetrically when the position of marker 20 changes. Opposing arrangements of deflectors can be used to improve the estimation accuracy of the marker distance by adjusting the distance between the intensity maxima of the reflected light. Analogous advantages in detection accuracy can also be achieved at the azimuth angle if the deflector structures are applied to marker 20 rotated by 90°.

[0034] The object position in space can also be extracted by using special patterns, as in the examples of Fig. 9, Fig. 10 and Fig. 11. In one embodiment, a pattern may, for example, consist of circular holographic deflectors 25 arranged concentrically. They deflect light in such a way that an intensity maximum for a specific wavelength is obtained at a specific distance. Fig. 10, for example, it can be seen that the holographic deflector 25 with the largest radius exhibits an interference pattern R with an interference maximum at a distance d3 of the detection device 30 from the marker 20. Likewise, it can be seen that the holographic deflector 25 with the medium radius exhibits an interference pattern R with an interference maximum at a distance d2 of the detection device 30 from the marker 20. Finally, it can be seen that the holographic deflector 25 with the smallest radius exhibits an interference pattern R with an interference maximum at a distance d1 of the detection device 30 from the marker 20. Fig. Figure 11 shows the corresponding wavelength-dependent intensity distributions P of the different wavelengths f1, f2, and f3 of the concentric holographic deflectors 25 as a function of the respective distances d1, d2, and d3, respectively. If the distance of the marker 20 to the detection device 30 is varied, this can be directly read in the image captured by the optoelectronic sensor 32 from the changing intensities of the rings and wavelengths.

[0035] In another implementation option, multiple markers 20 can be used in predefined spatial proximity to one another. For example, two, three, or more markers 20 can be attached to the bumper of a car, whose HOEs 21, 22, 23 can be designed with optical functions such that incident light of different wavelengths is fanned out in different spatial directions. If the markers are then detected, for example, by a second vehicle approaching from behind, the distance between the two vehicles can be encoded in the various markers 20 depending on the detected color(s). When the approaching vehicle approaches centrally and perpendicularly, both markers 20 change color simultaneously.

[0036] Problems in determining the position and location of markers 20 with HOEs 21, 22, and 23 can arise when the markers are detected at very low elevation angles. In this case, the effectively visible or detectable area approaches zero. Detection and evaluation of the reflection images becomes unreliable and, above a certain elevation limit, may no longer be possible. Fig. Figure 8 shows how such problems can be mitigated by special holographic structures 24 printed on the marker 20. By using holographic structures 24, for example, frame deflectors surrounding the QR code 10 that retroreflect light at very low elevation angles, a limiting alignment is indicated by the backscattered light. When the elevation angle reaches a critical range, the backreflected light from the frame deflector appears with particularly high intensity in the image captured by the optoelectronic sensor 32. This allows the detection device 30 to detect that the orientation of the marker 20 no longer allows a reliable location and / or position determination with sufficient certainty.

[0037] The orientation can be encoded through the intensity distribution and spectral composition of the holographic structures 24, such as a circumferential frame deflector. For example, the color of the frame deflector can change from green to red in the transition region to the critical area, or the reflected image of the frame deflector can change from a green rectangle to a red stop symbol.

[0038] Fig. Figure 6 schematically shows a flowchart of a localization method M for determining the position and orientation of an object marked with several optoelectronically detectable markers 20, such as a building, a machine, a land vehicle, a component, or a pressurizable surface. The localization method M can be implemented, for example, with a localization system 1 as shown in Fig. 3 can be realized as an example and for an application as in one of the Fig. 12 to 17 are used as examples.

[0039] In a first step M1, a time-resolved detection of light reflected by holographic optical elements 21, 22, 23 applied to the markers 20 is carried out by means of an optoelectronic sensor 32 of an optoelectronic detection device 30, the color depth of which is, for example, at least 8 bits. In a second step M2, a time-resolved sequence of two-dimensional reflection images is generated by the optoelectronic sensor 32 from the detected reflected light. In a step M3, an image processing device detects relative differences between the reflection behavior of a first of the optoelectronically detectable markers and the reflection behavior of a second of the optoelectronically detectable markers based on the spectrally resolved local intensity distribution in the sequence of two-dimensional reflection images detected by the optoelectronic sensor 32.This means that the intensities of various colors recorded in the reflection images are determined spatially resolved according to length and height in the reflection images, assigned to the respective different optoelectronically detectable markers, and the spatially resolved intensity distributions are arranged in a time-resolved sequence. This can be done, for example, by comparing the two-dimensional reflection images with a plurality of comparison and / or training data stored in a database for the captured two-dimensional reflection images. Finally, in a step M4, a position and orientation of the object relative to the optoelectronic detection device 30 can be determined by evaluating the detected differences.In particular, changes in the configuration, external shape or contour of the object can also be determined by this evaluation, since it is possible to discriminate between optoelectronically detectable markers attached at different locations on the object based on the relative differences in the local reflection behavior of the markers in the sequence of two-dimensional reflection images.

[0040] Optionally, in a further step M5, an object recognition of structures of a contrast code 10 of the markers 20 recorded in the two-dimensional reflection images captured by the optoelectronic sensor 32 can be carried out.

[0041] Fig. 12 shows a schematically illustrated diagram of a structure marked with several optoelectronically detectable markers 20, such as a bridge 100 and an associated optoelectronic detection device 30. The bridge 100 is provided with a number of markers 20 at various locations, which are as spatially evenly distributed as possible. In the example of the Fig. 12, six markers 20 are shown as being attached to the cheek surfaces of the supporting piers and the side of the bridge surface, although the number and exact attachment position of the markers 20 may, of course, differ from the illustrated example.

[0042] A detection device 30 can record a temporal sequence of reflection images in which light reflected by holographic optical elements applied to the various markers 20 can be seen in varying color and intensity distributions. Over time, changes in the statics and / or damage to the bridge 100 can lead to changes in the configuration or outer contour of the bridge elements in three-dimensional space. Due to the spatially and possibly temporally fixed attachment of the markers 20, this also leads to changes in the reflection behavior of individual markers relative to one another, so that the temporal sequence of reflection images reflects the temporal progression of the differences in the reflection behavior of individual markers relative to one another. The optoelectronically detectable markers 20 can be optimized in their optical function for the attachment location.For example, a first group of 20 markers can indicate large angular differences in the range of several degrees, while a second group of 20 markers can indicate fine angular changes in the range of a few arc minutes.

[0043] The structure 100 can be characterized as a reference point by a snapshot or continuous recording. For this purpose, the light returned by the markers 20 is spectrally evaluated. The markers 20 can be detected by a retroreflective function of the holographic optical elements or by illumination synchronized with the recording, for example, with lock-in technology or an external flash. If structural changes occur to the structure 100, the reflection behavior of the holographic optical elements of certain markers 20 changes in different ways, so that a comparison with the reference point measurement makes the structural changes recognizable.

[0044] Using a localization system 1 based on optoelectronically detectable markers 20, assessments of tunnels, dams, or bridge structures, for example, to detect structural damage, can be carried out cost-effectively, reliably, and with high precision. In particular, no complex visual inspection by experts is necessary; instead, a current status recording can provide direct information about the condition of the structure. Even the smallest changes can be made visible using optoelectronically detectable markers 20 with established hardware, such as cameras as optoelectronic detection devices 30. This allows highly accurate recordings of even small changes over time, for example, using computer-aided image analysis methods (computer vision, object recognition, machine learning, artificial intelligence).Structural changes in the building, such as deformations, cracks, or fractures, can also be detected with the localization system 1 without complex or costly sensors and without expensive or sensitive technology. Due to their simplicity, the optoelectronic detection devices 30 can also be installed in vehicles driving past buildings. The recorded images can be processed using appropriate algorithms and compared with images from earlier points in time.

[0045] Fig. 13 shows a schematically illustrated diagram of a land vehicle, such as a truck 200, marked with a plurality of optoelectronically detectable markers 20, and an associated optoelectronic detection device 30. The truck 200 is provided, for example, on the various wheels 201 with one of a number of markers 20. In the example of Fig. 13, two markers 20 are shown as being attached to the rear and front wheels 201, although the number and exact attachment position of the markers 20 may, of course, differ from the illustrated example.

[0046] A detection device 30 can record a temporal sequence of reflection images in which light reflected by holographic optical elements applied to the various markers 20 can be seen in varying color and intensity distributions. Over time, changes in the statics and / or damage to the vehicle axle can lead to a change in the relative position of the rear and front wheels. Due to the spatially and possibly temporally fixed attachment of the markers 20, this also leads to changes in the reflection behavior of individual markers relative to one another, so that the temporal sequence of reflection images reflects the temporal course of the differences in the reflection behavior of individual markers relative to one another. By translating the rotation angle information into a color value, a more distant evaluation sensor system can be implemented, which can determine the state of the object, such as the wheel position.

[0047] Fig. 14 shows a schematically illustrated diagram of a machine 300 marked with several optoelectronically detectable markers 20 and an associated optoelectronic detection device 30. The machine 300 has, for example, various gear elements, each of which is provided with one of a number of markers 20. In the example of Fig. 14, three markers 20 are shown as being attached to three gear elements, although the number and exact attachment position of the markers 20 may, of course, differ from the illustrated example.

[0048] A detection device 30 can record a temporal sequence of reflection images in which light reflected by holographic optical elements applied to the various markers 20 can be seen in varying color and intensity distributions. The detection device can measure multiple objects simultaneously, which enables their synchronous measurement. Over time, changes in the statics and / or damage to the gear elements and their axes can lead to a change in the relative position of the gear elements. Due to the spatially and possibly temporally fixed attachment of the markers 20, this also leads to changes in the reflection behavior of individual markers relative to one another, so that the temporal sequence of reflection images reflects the temporal course of the differences in the reflection behavior of individual markers relative to one another.However, the recording device can also precisely determine the current condition of the machine components.

[0049] Fig. 15 shows a schematically illustrated diagram of a machine 400 marked with at least one optoelectronically detectable marker 20 and an associated optoelectronic detection device 30. The machine 400 has, for example, an actuator-driven movable element 401, which is provided, for example, with one of a number of markers 20. In the example of Fig. 15, a marker 20 is shown as being attached to the actuator-driven movable element 401, wherein the number and exact attachment position of the markers 20 may, of course, differ from the illustrated example.

[0050] A detection device 30 can record a temporal sequence of reflection images in which light reflected by holographic optical elements applied to the various markers 20 can be seen in varying color and intensity distributions. During normal operation of the machine 400, a periodically recurring detection pattern can be detected in the temporal sequence of reflection images. Anomalies in this detection pattern can be identified as defects and assigned to a specific defect type, allowing targeted shutdown and / or maintenance of the machine 400 or the actuator-driven movable element 401.

[0051] Depending on the operating quality in a defect-free control state, the operation of an actuator-driven machine 400 is only associated with vibrations below a predetermined tolerance threshold. The detection pattern is deterministically reproducible and thus predictable. Due to the spatially and possibly temporally fixed attachment of the markers 20, the occurrence of an imbalance, material fatigue in moving machine elements such as drive belts or axle suspensions, or a change in the drive speed also leads to changes in the reflection behavior of individual markers relative to one another, so that the temporal sequence of reflection images reflects the temporal progression of the differences in the reflection behavior of individual markers relative to one another.

[0052] The detection of an impending defect in an actuator-driven machine 400 can be achieved, for example, by comparing a current reflection image with a reference reflection image during normal operation, for example, in time, frequency, or Laplace space. A common method for analyzing the frequency components of periodic signals is the Fourier transform, which can be directly applied here. Alternatively, changes in the color distribution in certain spectral ranges, or trends in deviations in the averaged color distribution or the averaged intensity distribution, can be detected.

[0053] The localization system 1 can also have self-learning properties: The anomaly that actually occurred is assigned to a reference anomaly after the repair or maintenance of the machine 400. This information is fed back into the analysis process of the localization system 1, which learns from the actual error and thus improves its error-specific detection in the future. Over time, an error signal database can be created that receives training data from all machines equipped with the localization system 1.

[0054] Fig. 16 shows a schematically illustrated diagram of a component marked with a plurality of optoelectronically detectable markers 20, such as a support beam 500, and an associated optoelectronic detection device 30. The support beam 500 is provided with a number of markers 20 at various attachment points, for example. In the example of Fig. 16, two markers 20 are shown as being attached to the support beam 500, although the number and exact attachment position of the markers 20 may, of course, differ from the illustrated example.

[0055] There are many applications where cost-effective monitoring of material changes or fractures can be beneficial, for example, in metal supports, protective coverings, reactors, boilers, gas vessels, pipelines, water reservoirs, pressure tanks, containers, and similar structural elements. Components 500 equipped with the localization system 1 can ensure early detection of damage and, subsequently, early correction, intervention, or repair of the detected defects in a cost-effective and reliable manner. Monitoring can be performed remotely and without contact and can be integrated into existing camera surveillance systems.

[0056] In some variants, optoelectronically detectable markers 20 can also be applied within the component 500 to fracture surfaces of predetermined breaking points, so that the markers 20 can only be detected when the component 500 is broken at the predetermined breaking point.

[0057] Fig. 17 is a schematically illustrated diagram of a pressurizable surface 600 marked with a plurality of optoelectronically detectable markers 20 and an associated optoelectronic detection device 30. The pressurizable surface 600 can, for example, be provided with a number of markers 20 at different attachment points. In the example of Fig.17, four markers 20 are shown as being applied to the pressurizable surface 600. The number and exact positioning of the markers 20 may, of course, differ from the illustrated example. Exposure can be either directed by a specified light source or by ambient light. In the latter case, a passive sensor can advantageously be defined, and a calibration of the holographic angle measurement can be performed under the given ambient light. This calibration may need to be repeated with changes in the ambient light.

[0058] The pressurizable surface 600 can be, for example, the surface of a container or a pipe. When the surface is subjected to changing pressure, the pressurized surface 600 will expand or compress. Based on the difference between the external and internal pressure of the pressurized surface 600, a deflection of the surface 600 results that is proportional to the pressure difference and depends on the surface strength, surface tension, surface elasticity, and temperature. This deflection of the pressurized surface 600 can be precisely determined based on the change in angle between the attached optoelectronically detectable markers 20 and the optoelectronic detection device 30.

[0059] Advantageously, several optoelectronic detection devices 30 can be used, which can be interconnected to form a measuring bridge, for example according to Wheatstone. This measure enables compensation for systematic errors, such as those caused by temperature fluctuations or contamination, at the analog level of the circuit. The same environmental influences act on all optoelectronically detectable markers 20, and thus, a temperature change or contamination essentially causes the same change in the voltage tapped at the optoelectronic detection devices 30 interconnected in a measuring bridge. At the output of the measuring bridge, these differences can then be compensated for as often as necessary via a reference calibration to the given lighting conditions and material properties of the pressurizable surface 600. Calibration is typically performed by passing through a characteristic curve, e.g.the artificial creation of all pressure levels in the intended measuring range.

[0060] Such non-contact measurement, which can be used across various pressure media, can detect impending changes in pressure situations, for example, in containers or lines, well in advance of defects or undesirable conditions occurring. Furthermore, potential hazards can be eliminated early through predictive maintenance. The non-contact measurement principle enables use in pressure media that are difficult to access, contaminated, or potentially explosive. Containers or lines can also be inspected externally for internal pressure conditions based on their surface.

[0061] The localization system 1 used on a pressurizable surface 600 ensures high sensitivity, as even thicker exterior walls are accessible for measurement due to the high angular selectivity of the holographic measurement principle. The localization system 1 can be used cost-effectively and across large areas, enabling field applications, such as gas pipelines or other critical infrastructure, for example, within an IoT monitoring network (smart sensor network). On the other hand, the high measurement accuracy enables applications under laboratory conditions, for example, in biological research, for example, to monitor the expansion of tissue.

[0062] In summary, the present invention relates to a concept for improved location and orientation determination of an object marked with an optoelectronically detectable marker. Holographic optical elements are incorporated into a two-dimensional, planar marker. These elements reflect ambient light or light from a dedicated additional light source with sufficiently high intensity, even at high scattering angles relative to the surface normal of the two-dimensional, planar marker. This enables very precise location determination of the marker with respect to all solid angles, since the holographic optical elements generate angle-dependent, distinguishable and linearly independent reflection patterns in an optoelectronic detection device. These patterns can be used to determine the position, location, and orientation of the marker and thus the corresponding position, location, and orientation of the object marked with it.

[0063] In particular, the concept for improved location and orientation determination can be applied to determining the position and orientation of objects by detecting relative differences in the reflection behavior of the holographic optical elements at various fixed attachment locations on or in the object when the respective position and orientation of the respective attachment locations, and thus of the entire object, changes. The markers according to the invention and their localization allow not only object recognition but also a more precise determination of the angular position, so that in addition to the absolute position of the localized object, the relative orientation of individual object components relative to other spatially distant object components of the same object can be determined.

[0064] Particularly in the case of machines, machine components, land vehicles, structures, building elements and / or pressurizable surfaces, such position and orientation determination can significantly facilitate the monitoring of operating conditions over time and, in particular, deviations from operating conditions classified as normal.

[0065] The concept can be supplemented by implementing special two-dimensional planar patterns and / or shapes of the holographic optical elements, so that the intensity of the reflected ambient light generates color-dependent intensity maxima depending on the distance of the marker from the optoelectronic detection device. Furthermore, special holographic optical deflection elements with retroreflectivity maxima at very high scattering angles relative to the surface normal of the two-dimensional planar marker can be provided, so that the detection of such intensity maxima in the optoelectronic detection device indicates that a critical angular range has been reached in which marker orientation can no longer be guaranteed with sufficient reliability due to the insufficient detectable marker area. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2021 / 0248338 A1

[0003] US 10 885 413 B1

[0003] DE 10 2017 218 544 A1

[0023]

Claims

[1] Localization system (1) for determining the position and / or orientation of an object (100; 200; 300; 400; 500; 600) marked with several optoelectronically detectable markers (20), comprising: an optoelectronic detection device (30) comprising: an optoelectronic sensor (32) which is designed to detect, in a time-resolved manner, light emanating from holographic optical elements (HOE; 21, 22, 23) applied to the markers (20), in particular diffracted and / or reflected and / or backscattered light, and to generate a time-resolved sequence of two-dimensional reflection images; and an image processing device (34) which is coupled to the optoelectronic sensor (32) and is designed to detect, on the basis of the spectrally resolved local intensity distribution of the sequence of two-dimensional reflection images detected by the optoelectronic sensor (32), relative differences between the reflection behavior of a first of the optoelectronically detectable markers (20) and the reflection behavior of a second of the optoelectronically detectable markers (20), and / or relative differences between the reflection behavior of a first of the optoelectronically detectable markers (20) in two temporally successive reflection images. [2] Localization system (1) according to claim 1, wherein the optoelectronic detection device (30) further comprises a memory (33) which is coupled to the image processing device (34) and stores a plurality of comparison and / or training data for the detected two-dimensional reflection images in a database. [3] Localization system (1) according to claim 1 or 2, wherein the image processing device (34) is further designed to carry out an object recognition of structures of a contrast code (10) of the markers (20) recorded in the two-dimensional reflection images captured by the optoelectronic sensor (32). [4] Localization system (1) according to one of claims 1 to 3, wherein at least one of the holographic optical elements (21, 22, 23) applied to the markers (20) has a holographic deflector (24) with high angular selectivity at low elevation angles relative to the surface of the marker (20). [5] Localization system (1) according to one of claims 1 to 4, wherein the color depth of the optoelectronic sensor (32) is at least 8 bits. [6] Localization system (1) according to one of claims 1 to 5, wherein at least one of the holographic optical elements (21, 22, 23) applied to the markers (20) has an optical function, so that a changing distance of the detection device (30) to the object (100; 200) is indicated by a color change and / or a change in the intensity distribution in the two-dimensional reflection images detected by the optoelectronic sensor (32). [7] Localization system (1) according to one of claims 1 to 6, wherein at least one of the holographic optical elements (21, 22, 23) applied to the markers (20) has an optical function, so that a changing object orientation for dedicated angular ranges is indicated by a color change and / or a change in the intensity distribution in the two-dimensional reflection images detected by the optoelectronic sensor (32). [8] Localization system (1) according to one of claims 6 and 7, wherein at least two of the holographic optical elements (21, 22, 23) applied to the markers (20) have duplicated optical functions for improving the reliability of a determination of the distance of the detection device (30) to the object (100; 200) and / or a determination of the object orientation for dedicated angular ranges. [9] Localization system (1) according to one of claims 1 to 8, wherein the holographic optical elements (21, 22, 23) applied to the markers (20) have a function that replicates a retroreflective function of classical optics. [10] Localization method (M) for determining the position and / or orientation of an object (100; 200; 300; 400; 500; 600) marked with several optoelectronically detectable markers (20), comprising the steps: time-resolved detection (M1) of light emanating from holographic optical elements (HOE; 21, 22, 23) applied to the markers (20), in particular diffracted and / or reflected and / or backscattered, by means of an optoelectronic sensor (32) of an optoelectronic detection device (30); generating (M2) a time-resolved sequence of two-dimensional reflection images by the optoelectronic sensor (32) from the detected reflected light; Detecting (M3) relative differences between the reflection behavior of a first of the optoelectronically detectable markers and the reflection behavior of a second of the optoelectronically detectable markers, and / or Detecting (M3) relative differences between the reflection behavior of a first of the optoelectronically detectable markers (20) in two temporally successive reflection images on the basis of the spectrally resolved local intensity distribution in the sequence of two-dimensional reflection images detected by the optoelectronic sensor (32). [11] Localization method (M) according to claim 10, wherein the evaluation (M3) of the spectral and / or local intensity distribution in the two-dimensional reflection images acquired by the optoelectronic sensor (32) comprises comparing the two-dimensional reflection images with a plurality of comparison and / or training data stored in a database for the acquired two-dimensional reflection images. [12] Localization method (M) according to one of claims 10 and 11, further comprising the steps: Carrying out (M5) an object recognition of structures of a contrast code (10) of the markers (20) recorded in the two-dimensional reflection images captured by the optoelectronic sensor (32). [13] Localization method (M) according to one of claims 10 to 12, wherein at least one of the holographic optical elements (21, 22, 23) applied to the markers (20) has a holographic deflector (24) with high angular selectivity at low elevation angles relative to the surface of the marker (20). [14] Localization method (M) according to one of claims 10 to 13, wherein the color depth of the optoelectronic sensor (32) is at least 8 bits. [15] Localization method (M) according to one of claims 10 to 14, wherein at least one of the holographic optical elements (21, 22, 23) applied to the markers (20) has an optical function, so that a changing distance of the detection device (30) to the object (100; 200) is indicated by a change in the spectral and / or local intensity distribution in the two-dimensional reflection images detected by the optoelectronic sensor (32). [16] Localization method (M) according to one of claims 10 to 15, wherein at least one of the holographic optical elements (21, 22, 23) applied to the markers (20) has an optical function, so that a changing object orientation for dedicated angular ranges is indicated by a change in the spectral and / or local intensity distribution in the two-dimensional reflection images detected by the optoelectronic sensor (32). [17] Localization method (M) according to one of claims 15 and 16, wherein at least two of the holographic optical elements (21, 22, 23) applied to the markers (20) have duplicated optical functions for improving the reliability of a determination of the distance of the detection device (30) to the object (100; 200) and / or a determination of the object orientation for dedicated angular ranges. [18] Localization method (M) according to one of claims 10 to 17, wherein the holographic optical elements (21, 22, 23) applied to the markers (20) have a function that replicates a retroreflective function of classical optics.

Images