Localization system and localization method with markers having holographic optical elements

Abstract

The invention discloses a concept for improved time-resolved location and / or position determination of an object marked with several optoelectronically detectable markers, in which holographic optical elements are introduced into two-dimensionally planar markers, at which ambient light is reflected back with sufficiently high intensity even at high scattering angles relative to the surface normal of the two-dimensionally planar markers.For this purpose, it is proposed to detect light reflected by holographic optical elements (HOE; 21, 22, 23) applied to the marker (20) in an optoelectronic sensor (32) and to generate a temporal sequence of two-dimensional reflection images, and to determine in an image processing device (34) a time-resolved position and / or orientation of the object relative to the optoelectronic detection device (30) or the environment marked with detectable markers (20) by evaluating the spectral and local intensity distributions in the two-dimensional reflection images detected by the optoelectronic sensor (32).

Description

Technical area

[0001] The present invention relates to a localization system and a localization method based on angle-dependent reflection measurements of holographic optical elements as part of markers to be located. State of the art

[0002] Various positioning methods can be used to determine the position and / or orientation of objects. Optical object recognition, for example, often uses high-contrast geometric structures that can be recognized easily, precisely, and reliably. Examples include optoelectronically detectable color-contrasting markings consisting of differently sized geometric symbols or shapes, such as lines or polygons, with gaps in between, so that the luminance difference at the boundaries between shapes and gaps is as high as possible. Such markings can have one-dimensional coding (e.g., barcodes) or two-dimensional coding (e.g., DataMatrix codes, QR codes, or Aztec codes), each with a different useful information density.The data encoded in a marker can be machine-readable and further processed electronically using optical readers such as cameras or camera scanners.

[0003] Markers can be equipped with holographic optical elements (HOEs) to increase information density. Examples of such markers are disclosed in US 2021 / 0248338 A1, PT 99405 A, CZ 2004-1136 A3, and US 10 885 413 B1. Disclosure of the invention

[0004] The present invention provides a localization system and a localization method, each having the features of the independent claims. Further advantageous embodiments are the subject of the dependent claims.

[0005] A localization system for the time-resolved location and position determination of an object marked with multiple optoelectronically detectable markers comprises an optoelectronic detection device. This device has an optoelectronic sensor and an image processing device coupled to the optoelectronic sensor. The optoelectronic sensor is designed to detect, in a time-resolved manner, light emanating from holographic optical elements applied to the markers, in particular diffracted and / or reflected and / or backscattered light, and reflected by the holographic optical element onto the optoelectronic sensor, and to generate a time-resolved sequence of two-dimensional reflection images.The image processing device is designed to determine a position and orientation of the object relative to the optoelectronic detection device by evaluating the spectrally resolved local intensity distribution in the two-dimensional reflection image detected by the optoelectronic sensor.

[0006] A localization method for the time-resolved location and position determination of an object marked with a plurality of optoelectronically detectable markers comprises the steps of: time-resolved detection of light emanating from holographic optical elements applied to the markers, in particular diffracted and / or reflected and / or backscattered light, and reflected by the holographic optical element onto the optoelectronic sensor by means of an optoelectronic sensor of an optoelectronic detection device; generating a time-resolved sequence of two-dimensional reflection images by the optoelectronic sensor from the detected reflected light; evaluating the spectrally resolved local intensity distribution in the two-dimensional reflection images detected by the optoelectronic sensor;and determining a time series of positions and orientations of the object relative to the optoelectronic detection device on the basis of the evaluated spectrally resolved local intensity distribution;

[0007] A navigation method for a land, water, or air vehicle marked with multiple optoelectronically detectable markers comprises the steps of: time-resolved detection of light emanating from holographic optical elements applied to the markers, in particular diffracted and / or reflected and / or backscattered light, and reflected by the holographic optical element onto the optoelectronic sensor by means of an optoelectronic sensor of an optoelectronic detection device; generating a time-resolved sequence of two-dimensional reflection images by the optoelectronic sensor from the detected reflected light; determining a navigation path of the land, water, or air vehicle relative to the optoelectronic detection device on the basis of the time-resolved sequence of two-dimensional reflection images;and preferably generating navigation control information for the land, water or air vehicle for adapting the determined navigation path, wherein the navigation control information is dependent on an optical function of the holographic optical elements applied to at least one of the markers;

[0008] Holographic optical elements (HOEs) are optically active structures whose functional principle is based on holography. In an HOE, the imaging properties of a hologram are utilized, or conventional optics are implemented using a hologram, so that light waves can be deflected, split, focused, and / or expanded. HOEs can also be formed, for example, from extremely thin film foils that can decompose incident light into its spectral colors. However, only the light rays that fall within a specific wavelength range are diffracted. By using HOEs, diffraction maxima can be generated, which arise due to constructive interference of light waves at the HOEs.

[0009] In contrast to conventional optics, HOEs deflect light not through refraction, but through diffraction at the volume grating. HOEs can be manufactured for both transmission and reflection, enabling new designs through a free choice of angles of incidence and reflection, or diffraction angle. The holographic diffraction grating is, for example, exposed into a thin, photosensitive film. Volume diffraction can also give a HOE a characteristic wavelength and angle selectivity or filter function. Depending on the imaging condition, such as a wavelength or angle, only light from defined directions and with defined wavelengths is diffracted by the structure of the HOE. Advantages of the invention

[0010] According to some embodiments of the localization system and the localization method, the optoelectronic detection device may further comprise a memory coupled to the image processing device and storing a plurality of comparison and / or training data for the acquired two-dimensional reflection images in a database. This enables rapid and reliably reproducible localization of the object from known reference data.

[0011] According to various embodiments of the localization system and the localization method, the image processing device can further perform object recognition of structures of a contrast code of the markers recorded in the two-dimensional reflection images captured by the optoelectronic sensor. The combination of contrast codes, such as QR codes, with holographic optical elements enables the implementation of redundant recognition mechanisms, which improves robustness against sources of interference, reduces susceptibility to misdeterminations, and increases the accuracy of position and distance determination.

[0012] According to several embodiments of the localization system and the localization method, at least one of the holographic optical elements applied to the markers can have a holographic deflector with high angular selectivity at low elevation angles relative to the surface of the marker. This enables the implementation of a warning function in case of insufficient detectable effective area of ​​the marker in unfavorable orientations.

[0013] According to some embodiments of the localization system and the localization method, the color depth of the optoelectronic sensor can be at least 8 bits. This advantageously increases the spatial and angular resolution of the position and orientation determination.

[0014] According to some embodiments of the localization system and the localization method, at least one of the holographic optical elements applied to the markers can have an optical function, so that a changing distance of the detection device from the object is indicated by a color change and / or a change in the intensity distribution in the two-dimensional reflection images captured by the optoelectronic sensor. This allows for more precise distance determination.

[0015] According to some embodiments of the localization system and the localization method, at least one of the holographic optical elements applied to the markers can have an optical function, so that a changing object orientation for dedicated angular ranges is indicated by a color change and / or a change in the intensity distribution in the two-dimensional reflection images captured by the optoelectronic sensor. This allows for a more precise determination of the object orientation.

[0016] According to some embodiments of the localization system and the localization method, at least two of the holographic optical elements applied to the markers can have duplicated optical functions to improve the reliability of determining the distance of the detection device to the object and / or determining the object orientation for dedicated angular ranges. This advantageously enables particularly robust distance and object orientation determination, since the evaluated back reflections of the at least two holographic optical elements can be mutually validated.

[0017] The above embodiments and developments can be combined with one another as desired, where appropriate. Further embodiments, developments, and implementations of the invention also include combinations of features of the invention not explicitly mentioned above or described below with respect to the exemplary embodiments. In particular, those skilled in the art will also add individual aspects as improvements or additions to the respective basic forms of the invention. Short description of the drawings

[0018] Further features and advantages of the invention are explained below with reference to the figures. Fig. 1: an exemplary illustration of an optoelectronically detectable marker in the form of a QR code; Fig. 2: a schematic illustration of an optoelectronically detectable marker in the form of a QR code with holographic optical elements superimposed on the coding according to an embodiment; Fig. 3: a schematic illustration of the optoelectronic detection of the marker of the Fig. 2 using a localization system according to an embodiment; Fig. 4: an exemplary diagram of a two-dimensionally resolved optoelectronic measurement signal distribution of the holographic optical elements of the marker of the Fig. 2 in a localization system of Fig. 3; Fig. 5: another exemplary diagram of a two-dimensionally resolved optoelectronic measurement signal distribution of the holographic optical elements of the marker of the Fig. 2 in a localization system of Fig. 3; Fig. 6: a flowchart of a localization method for time-resolved location and position determination of an object marked with several optoelectronically detectable markers according to one embodiment; Fig. 7: exemplary structures for redundant holographic optical elements on an optoelectronically detectable marker according to an embodiment; Fig. 8: a schematic illustration of an optoelectronically detectable marker in the form of a QR code with holographic optical elements superimposed on the coding and a holographic deflector according to a further embodiment; Fig. 9: a schematic illustration of an optoelectronically detectable marker in the form of a QR code with holographic optical elements superimposed on the coding according to a further embodiment; Fig. 10: a schematic plot of the reflection intensity maxima along the distance from the optoelectronically detectable marker of the Fig. 9; Fig. 11: a schematic plot of the frequency-dependent reflection intensity along the distance from the optoelectronically detectable marker of the Fig. 9; Fig. 12: a schematically illustrated diagram of an aircraft marked with a plurality of optoelectronically detectable markers and an associated optoelectronic detection device according to an embodiment; Fig. 13: a schematically illustrated diagram of a watercraft marked with multiple optoelectronically detectable markers and an associated optoelectronic detection device according to one embodiment; and Fig. 14: a schematic plot of a time series of position and orientation vectors determined for an object marked with several optoelectronically detectable markers. Description of embodiments

[0019] Fig. Figure 1 shows an exemplary illustration of an optoelectronically detectable marker in the form of a QR code 10. The exemplary QR code 10 is a deliberately constructed pattern of high-contrast geometric structures. The polygons are optoelectronically detectable and color-contrasted with gaps in between, so that the luminance difference at the boundaries between shapes and gaps is as high as possible. Such markings can have one-dimensional coding (e.g., barcodes) or two-dimensional coding (e.g., DataMatrix codes, QR codes, or Aztec codes), each with a different payload density. The data encoded in such a marker can be machine-readable using optical reading devices, such as cameras or camera scanners, and further processed electronically.

[0020] Determining the location and / or orientation of objects marked by such optoelectronically detectable markers can be error-prone and often inaccurate. The accuracy of the location and / or orientation determination often varies with the distance to the marked object and the lighting conditions in the respective environment. Determining the angular orientation, i.e., the spatial alignment to the optical reader, can also be severely impaired. As distances increase, higher-resolution optoelectronic detection devices may be required. Detecting and locating the patterns on such markers requires the use of complex object recognition algorithms that require high computing power.

[0021] The use of markers equipped with holographic optical elements (HOEs) can improve the localization of the objects marked with them. In particular, but not exclusively, position detection can be significantly improved through optimized angular orientation determination. Better angular resolution, potentially better spatial resolution, lower costs due to lower camera resolution, computationally efficient and fast position determination, as well as possible extensions of existing localization solutions can be achieved through the use of optoelectronically detectable markers in the form of one- or two-dimensional, flat contrast codes such as barcodes, DataMatrix codes, QR codes, or Aztec codes, with holographic optical elements (HOEs) superimposed on the code.

[0022] Fig. Figure 2 shows an example of such an optoelectronically detectable marker 20 with a contrast coding, here a QR code 10. Overlaid on the QR code 10 are HOEs 21, 22 and 23, which are spatially applied to the marker 20 at predefined locations along the surface area of ​​the QR code 10. The number, size, shape, type and position of the HOEs 21, 22 and 23 is shown in Fig. 2 is shown only as an example. It is possible to provide more or fewer than three HOEs 21, 22 and 23 with a different size, shape and / or position on the marker 20.

[0023] The optical function of a hologram is created by the diffraction of incident optical wavefronts by a diffraction grating in the volume of the holographic film. The diffraction grating was previously exposed into the volume of the holographic film during an exposure process. The diffraction grating forms according to an interference pattern generated by two optical recording wavefronts as a modulation of the refractive index in the holographic film. The interference pattern is generated by the superposition of two coherent recording waves in the volume of the holographic film. The interference of the two recording wavefronts creates an interference grating in the volume of the holographic film, which is permanently imprinted into the volume of the holographic film by the exposure process. The parameters of the exposure process (wavefront shape, spectral properties, intensity distribution, and duration) determine the properties of the generated diffraction grating.HOEs can be produced, for example, as hologram structures by holographic wavefront printing, as taught in the document DE 10 2017 218 544 A1.

[0024] Fig. Figure 3 shows a schematic illustration of the optoelectronic detection of the marker 20 of the Fig. 2 using a localization system 1. The localization system 1 comprises an optoelectronic detection device 30, such as a video camera, a camera scanner, or a similar device. Ambient light emitted and reflected by the marker 20 and in particular the HOEs 21, 22, and 23 is recorded via suitable detection optics 31 and fed to an optoelectronic sensor 32. The optoelectronic sensor 32 can, for example, be a CCD sensor, a CMOS sensor, an active pixel sensor (APS), a photodiode array, or another suitable electromagnetic radiation detector with spatial resolution.

[0025] The image captured by the optoelectronic sensor 32 is transferred to an image processing device 34, which evaluates the captured image and transmits the evaluation result to an output interface 35 of the capture device 35. The image processing device 34 can be, for example, a microprocessor, a graphics processor, an ASIC, an FPGA, or any other suitable computing device. The output interface 35 can be, for example, a display or another user interface. It may also be possible for the output interface 35 to establish a communication connection to another device, to which the evaluation result of the image processing device 34 is transmitted via the output interface 35.

[0026] The optoelectronic detection device 30 may also have a memory 33, such as a ROM, a RAM, or another type of data storage, in which, on the one hand, configuration parameters and operating software for operating the detection device 35 and, in particular, the image processing device 34 are stored. On the other hand, comparison and / or training data can be stored in the memory 33 in a database that enables a comparison of the image captured by the optoelectronic sensor 32 with known images or optoelectronic detection signatures of markers in different positions, orientations, and / or locations.

[0027] The optoelectronic detection device 30 serves, on the one hand, for the general identification and localization of an object marked with a marker 20 based on the contrast coding, such as the QR code 10. For this purpose, the image processing device 34 performs object recognition of the structures of the QR code 10 captured in the image captured by the optoelectronic sensor 32 and compares them with known QR codes, for example, by comparing them with known QR codes stored in a database of the memory 33. Subsequently, the spatial orientation can be deduced by determining the size, distance, and mutual orientation of the patterns and structures of the QR code 10.

[0028] Additionally or alternatively, the image captured by the optoelectronic sensor 32 may include back-reflected light from the HOEs 21, 22, and 23. In addition to the geometric position of the structure of the HOEs 21, 22, and 23, the position and orientation of the object relative to the optoelectronic detection device 30 may be encoded by the color of the back-scattered light. Fig. 4 and Fig. 5 show two exemplary diagrams of two-dimensionally resolved optoelectronic measurement signal distributions P1 and P2 of the HOEs 21, 22 and 23 of the marker 20 of the Fig. 2 by an optoelectronic detection device 30 of the localization system 1 of the Fig. 3.

[0029] HOEs are structures written into a photo-optical layer that redirect incident light. Depending on how the HOE's structure is designed, its optical function can be determined, for example, a scattering hologram, a retroreflector, a mirror, a concave mirror, or a lens. HOEs can also exhibit high angular or wavelength selectivity in their function. For example, the components of white light are scattered to different degrees in different spatial directions. Therefore, the orientation of the HOE relative to the illumination source can be encoded by the backscattered wavelength and determined using the image captured by the optoelectronic sensor 32 of an optoelectronic detection device 30.The spectrally resolved spatial intensity distribution P of three back reflections A, B and C of different HOEs 21, 22 and 23 can be bijectively mapped onto the spatial function F of the HOEs 21, 22 and 23 on a marker 20 measured at azimuth α and elevation β relative to the surface of the marker 20. For example, in . Fig. 4 from the spectrally resolved spatial intensity distribution P1[I(A), I(B), I(C)] of the location-dependent intensities and colors on the two-dimensional image of the optoelectronic sensor 32, the spatial function F(α1, β1) can be deduced. Analogously, for example, in Fig. 5 From the spectrally resolved local intensity distribution P2[I(A),I(B), I(C)] of the location-dependent intensities and colors on the two-dimensional image of the optoelectronic sensor 32, the spatial function F(α2, β2) can be inferred.

[0030] An optoelectronic sensor 32 of an optoelectronic detection device 30 can, for example, have a color depth of at least 8 bits. This means that the optoelectronic sensor 32 can distinguish between approximately 16 million color nuances. This allows the optoelectronic detection device 30 to perform very precise angle measurements of both the azimuth angle α and the elevation angle β. Because several differently positioned HOEs 21, 22, and 23 on the marker 20 each generate unique, linearly independent color patterns on the optoelectronic sensor 32, the respective associated angular orientation of the marker 20 can be uniquely encoded. If the orientation of the object or the position of the optoelectronic detection device 30 relative to the object changes and thus also the orientation of the marker 20 relative to the optoelectronic detection device 30, the wavelengths of the light backscattered by the HOEs 21, 22 and 23 shift.This also allows conclusions to be drawn about the change in position.

[0031] The localization system may additionally comprise a light source 50, which may be provided as a separate device or as part of the detection device 30.

[0032] For example, the light source 50 can emit white light W in a targeted manner onto the marker 20, so that the position of the illumination and recording optics is known a priori. This makes determining the location and angular orientation of the marker 20 based on the back-reflection patterns in the image of the optoelectronic sensor 32 easier and more reliable. Furthermore, the light emitted by the light source 50 can be modulated, allowing a lock-in mechanism to achieve a clear assignment of the backscattered light and an improvement in the signal-to-noise ratio.

[0033] HOEs 21, 22, and 23 can have optimized optical functions, such as retroreflectivity with a limited angular range, scattering functions with high angular selectivity, or any combination of these functions. Furthermore, HOEs 21, 22, and 23 can be designed for light outside the visible range, such as in the UV or infrared range.

[0034] Fig. Figure 7 shows, by way of example, that special patterns and / or special color combinations of HOEs can be used, so that the distance determination of the detection device 30 to the marker 20 can be improved via pattern recognition with regard to color, shape, and size. Due to the angular selectivity of holograms, a change in angle or the object orientation in space can be precisely encoded. For example, one and the same optical function can be applied multiple times to the marker surface. The alignment of wavelengths and intensities of the redundant optical functions can implement a control functionality. Alternatively, the optical function of the surfaces can also be selected such that the intensities of the back-diffracted light must differ. The deflection function can, for example, also be designed in the opposite direction, resulting in a point-symmetric intensity distribution of the back-diffracted light.If the position of marker 20 relative to the detection device 30 changes, this distribution must also change point-symmetrically. Areas addressing different wavelengths can be arranged symmetrically or asymmetrically, whereby the color composition of the reflected light also changes symmetrically or asymmetrically when the position of marker 20 changes. Opposing arrangements of deflectors can be used to improve the estimation accuracy of the marker distance by adjusting the distance between the intensity maxima of the reflected light. Analogous advantages in detection accuracy can also be achieved at the azimuth angle if the deflector structures are applied to marker 20 rotated by 90°.

[0035] The object position in space can also be extracted by using special patterns, as in the examples of Fig. 9, Fig. 10 and Fig. 11. In one embodiment, a pattern may, for example, consist of circular holographic deflectors 25 arranged concentrically. They deflect light in such a way that an intensity maximum for a specific wavelength is obtained at a specific distance. Fig. 10, for example, it can be seen that the holographic deflector 25 with the largest radius exhibits an interference pattern R with an interference maximum at a distance d3 of the detection device 30 from the marker 20. Likewise, it can be seen that the holographic deflector 25 with the medium radius exhibits an interference pattern R with an interference maximum at a distance d2 of the detection device 30 from the marker 20. Finally, it can be seen that the holographic deflector 25 with the smallest radius exhibits an interference pattern R with an interference maximum at a distance d1 of the detection device 30 from the marker 20. Fig. Figure 11 shows the corresponding wavelength-dependent intensity distributions P of the different wavelengths f1, f2, and f3 of the concentric holographic deflectors 25 as a function of the respective distances d1, d2, and d3, respectively. If the distance of the marker 20 to the detection device 30 is varied, this can be directly read in the image captured by the optoelectronic sensor 32 from the changing intensities of the rings and wavelengths.

[0036] In another implementation option, multiple markers 20 can be used in predefined spatial proximity to one another. For example, two, three, or more markers 20 can be attached to the bumper of a car, whose HOEs 21, 22, 23 can be designed with optical functions such that incident light of different wavelengths is fanned out in different spatial directions. If the markers are then detected, for example, by a second vehicle approaching from behind, the distance between the two vehicles can be encoded in the various markers 20 depending on the detected color(s). When the approaching vehicle approaches centrally and perpendicularly, both markers 20 change color simultaneously.

[0037] Problems in determining the position and location of markers 20 with HOEs 21, 22, and 23 can arise when the markers are detected at very low elevation angles. In this case, the effectively visible or detectable area approaches zero. Detection and evaluation of the reflection images becomes unreliable and, above a certain elevation limit, may no longer be possible.

[0038] Fig. Figure 8 shows how such problems can be mitigated by special holographic structures 24 printed on the marker 20. By using holographic structures 24, for example, frame deflectors surrounding the QR code 10 that retroreflect light at very low elevation angles, a limiting alignment is indicated by the backscattered light. When the elevation angle reaches a critical range, the backreflected light from the frame deflector appears with particularly high intensity in the image captured by the optoelectronic sensor 32. This allows the detection device 30 to detect that the orientation of the marker 20 no longer allows a reliable location and / or position determination with sufficient certainty.

[0039] The orientation can be encoded through the intensity distribution and spectral composition of the holographic structures 24, such as a circumferential frame deflector. For example, the color of the frame deflector can change from green to red in the transition region to the critical area, or the reflected image of the frame deflector can change from a green rectangle to a red stop symbol.

[0040] Fig. Figure 6 schematically shows a flowchart of a localization method M for the time-resolved location and position determination of an object marked with several optoelectronically detectable markers 20, such as an aircraft during a takeoff or landing procedure or a watercraft during a departure or docking maneuver. The localization method M can be implemented, for example, with a localization system 1 as shown in Fig. 3 can be realized as shown in example.

[0041] In a first step M1, a time-resolved detection of light reflected by holographic optical elements HOE; 21, 22, 23 applied to the markers 20 is carried out by means of an optoelectronic sensor 32 of an optoelectronic detection device 30, whose color depth is, for example, at least 8 bits. In a second step M2, a time-resolved sequence of two-dimensional reflection images is generated by the optoelectronic sensor 32 from the detected reflected light. In a step M3, an image processing device evaluates the spectrally resolved spatial intensity distribution in the two-dimensional reflection images captured by the optoelectronic sensor 32. This means that the intensities of various colors recorded in the reflection images are determined spatially resolved according to length and height in the reflection images, and the spatially resolved intensity distributions are arranged in a time-resolved sequence.This can be done, for example, by comparing the two-dimensional reflection images with a plurality of comparison and / or training data stored in a database for the acquired two-dimensional reflection images. Finally, in a step M4, a time series of positions and orientations of the object relative to the optoelectronic detection device 30 can be determined based on the spectrally resolved spatial intensity distribution evaluated in step M3.

[0042] Optionally, in a further step M5, an object recognition of structures of a contrast code 10 of the markers 20 recorded in the two-dimensional reflection images captured by the optoelectronic sensor 32 can be carried out.

[0043] Similarly, a navigation method M for a land, water or air vehicle marked with several optoelectronically detectable markers 20, such as those described in the Fig. 12 and Fig. 13 shown aircraft 100 or ships 200. The navigation method M can be carried out, for example, with the aid of a localization system 1 as in Fig. 3 can be realized as shown in example.

[0044] In a first step M1, a time-resolved detection of light reflected by holographic optical elements HOE; 21, 22, 23 applied to the markers 20 is performed by an optoelectronic sensor 32 of an optoelectronic detection device 30. In a second step M2, a time-resolved sequence of two-dimensional reflection images is generated from the detected reflected light by the optoelectronic sensor 32. Subsequently, in a step M4, a navigation path of the land, water, or aircraft 100; 200 relative to the optoelectronic detection device 30 is determined based on the time-resolved sequence of two-dimensional reflection images.

[0045] To adapt the determined navigation path, navigation control information for the land, water, or air vehicle 100; 200 is then generated, wherein the navigation control information is dependent on an optical function of the holographic optical elements 21, 22, 23 applied to at least one of the markers 20. For example, the optical function of the holographic optical elements 21, 22, 23 can encode a color change, a color intensity change, a reflection pattern change, or another discrete jump in a reflection parameter of the holographic optical elements 21, 22, 23 detectable in the two-dimensional reflection images, based on the occurrence of which a deviation from a desired navigation path of the land, water, or air vehicle 100; 200 can be determined.The generated navigation control information can be used to indicate a departure from the desired navigation path and / or to further generate navigation control signals for actively correcting the navigation path of the land, water or air vehicle 100; 200.

[0046] Fig. 12 shows a schematic illustration of an aircraft, shown here as an example as an aircraft 100 in a front view, which is in the approach to a runway 101 of an airport. The aircraft 100 is provided with a number of markers 20 at various, as evenly distributed as possible, locations. In the example of Fig. 12, three markers 20 are shown as being attached to the underside of the fuselage and the wing tips, although the number and exact attachment position of the markers 20 may, of course, differ from the illustrated example.

[0047] A capture device 30 can record a temporal sequence of reflection images in which light reflected by holographic optical elements applied to the various markers 20 can be seen in varying color and intensity distributions. During a takeoff or landing of the aircraft 100, the aircraft 100 moves in three-dimensional space, and thus also the markers 20, so that the temporal sequence of reflection images reflects the temporal progression of the reflection changes.

[0048] It may be possible to provide other objects, preferably static objects relative to the aircraft 100, such as a runway or runways of an airport, with markers 20 so that their holographic optical elements also reflect light, which can serve as a reference relative to the moving object on the captured temporal sequence of reflection images.

[0049] In one embodiment, the optoelectronic detection device 30 can be located on board the vehicle 100. In this case, the static markers, e.g., those placed in the surrounding area or on the runway, are used for relative location and orientation determination. The determined location and orientation information can be used directly to control the vehicle 100.

[0050] Fig. Figure 13 shows a schematic illustration of a watercraft, shown here as a ship 200 in front view, which is moored at a pier 201 of a harbor. The ship 200 is provided with a number of markers 20 at various, as evenly distributed as possible, locations. In the example of Fig. 13, three markers 20 are shown as being attached to the upper deck and the various bow sides, although the number and exact attachment position of the markers 20 may of course differ from the illustrated example.

[0051] A detection device 30 can record a temporal sequence of reflection images in which light reflected by holographic optical elements applied to the various markers 20 can be seen in varying color and intensity distributions. During a departure or docking maneuver of the ship 200, the ship 200 moves in three-dimensional space, and thus also the markers 20, so that the temporal sequence of reflection images reflects the temporal course of the reflection changes.

[0052] It may be possible to provide other objects, preferably static objects in relation to the ship 200, such as a pier of a harbor, with markers 20, so that their holographic optical elements also reflect light, which can serve as a reference relative to the moving object on the recorded temporal sequence of reflection images.

[0053] In one embodiment, the optoelectronic detection device 30 can be located on board the vehicle 200. In this case, the static markers, e.g., those attached in the surrounding area or to the piers, are used for relative location and orientation determination. The determined location and orientation information can be used directly on board the vehicle 200 for control or tracking without prior transmission.

[0054] Fig. Figure 14 shows a schematic plot of a time series of position and orientation vectors Q, which were determined for an object marked with several optoelectronically detectable markers. The plot can be used, for example, for position and orientation vectors Q of an aircraft 100, as exemplified in Fig. 12 or a ship 200 as exemplified in Fig. 13 shown apply.

[0055] At a first time t1, a spectrally resolved spatial intensity distribution F(α1, β1, γ1, x1, y1, z1) results on the respective reflection image of the temporal sequence of reflection images, which depends on the Euler angles α1, β1, γ1 characterizing the spatial orientation of the object and the coordinates x1, y1, z1 characterizing the position. If the object and thus the markers 20 continue to move with the object in three-dimensional space, a spectrally resolved local intensity distribution F(α2, β2, γ2, x2, y2, z2) results at a second time t2 following the first time t1, which is dependent on the Euler angles α2, β2, γ2 characterizing the spatial position and orientation of the object and the coordinates x2, y2, z2 characterizing the position.Similarly, at a third time t3 following the second time t2, a spectrally resolved spatial intensity distribution F(α3, β3, γ3, x3, y3, z3) results, which depends on the Euler angles α3, β3, γ3 characterizing the spatial position and orientation of the object and the coordinates x3, y3, z3 characterizing the position.

[0056] From the spectrally resolved local intensity distributions F(α1, β1, γ1), F(α2, β2, γ2) and F(α3, β3, γ3), a time series of positions and orientations of the object - for example, the aircraft 100 or the ship 200 - relative to the optoelectronic detection device 30 can then be calculated.

[0057] In a navigation method for a land, water, or air vehicle, such as the aircraft 100 or the ship 200, in addition to the pure time-resolved determination of the position and orientation of the vehicle, predetermined information about the navigation path or its correction can already be encoded in optical functions of the holographic optical elements of some or all of the markers 20 used. This allows information about the quality of the navigation path to be carried along with the vehicle without having to transmit such navigation information separately to the optoelectronic detection device 30.

[0058] The localization method M of the Fig.6 is performed iteratively to determine a navigation path of a vehicle 100 or 200, which is marked with markers 20. The actual navigation path then depends on the time series of positions and orientations of the vehicle 100 or 200 determined using the localization method M.

[0059] In order to be able to adapt or correct the determined actual navigation path, certain optical functions of the holographic optical elements applied to at least one of the markers 20 are utilized. These optical functions are reflected in the spectrally resolved spatial intensity distribution, so that navigation control information for the vehicle 100 or 200 can be generated depending on the optical functions.

[0060] For example, a ship 200 can be equipped with markers 20 that reflect light in a specific spectral range as long as the predetermined navigation path is maintained, for example, a correct entry into an anchorage in a harbor. However, if the predetermined navigation path is departed, the spectral reflectance behavior of certain markers 20 changes, so that the optoelectronic detection device 30 can generate corresponding navigation control information about the departure from the predetermined navigation path based on the changed spectral reflectance behavior of the markers 20, without requiring detailed information about the navigation path itself.

[0061] By using a suitable number of independent markers 20 and / or implementing separate optical functions on different markers 20, any number of discriminatory features for deviations from navigation paths can be created. This allows for higher-dimensional navigation to be implemented, for example, across a multitude of independent markers, spectral reflection ranges, and / or intensity distributions.

[0062] One of the advantages of such a navigation method is that the specific navigation path does not need to be known to the optoelectronic detection device 30, but only information about the assignment of changes in the spectral reflection behavior of individual markers needs to be transmitted to the optoelectronic detection device 30. This significantly reduces the storage requirements and / or transmission capacity of the detection device 30.

[0063] In addition, a simple control scheme can be implemented without complex signal analysis, which is very cost-effective and can be implemented in real-time or near real-time, since complex image processing and analysis are eliminated. The optoelectronic detection system can also be located on board the vehicle 100 or 200 if a specific environment in which navigation is to be carried out is equipped with corresponding markers 20.

[0064] The principle of using optical functions in temporally resolved reflection distributions of optoelectronically detectable markers can also be used for tracking applications. For example, markers 20 on aircraft 100 approaching a runway under specified target approach parameters can provide indications of deviations from the target approach parameters.

[0065] In summary, the present invention relates to a concept for improved location and orientation determination of an object marked with an optoelectronically detectable marker. Holographic optical elements are incorporated into a two-dimensional, planar marker. These elements reflect ambient light or light from a dedicated additional light source with sufficiently high intensity, even at high scattering angles relative to the surface normal of the two-dimensional, planar marker. This enables very precise location determination of the marker with respect to all solid angles, since the holographic optical elements generate angle-dependent, distinguishable and linearly independent reflection patterns in an optoelectronic detection device. These patterns can be used to determine the position, location, and orientation of the marker and thus the corresponding position, location, and orientation of the object marked with it.

[0066] In particular, the concept for improved location and attitude determination can be applied to the path control of moving objects, for example, as a basis for improving complex navigation maneuvers in which high-dimensional localization and control tasks must be mastered. The markers according to the invention and their localization allow not only object detection but also a more precise determination of the angular position, so that in addition to the absolute position of the localized object, the relative object orientation relative to, for example, stationary reference structures in the environment can be determined.

[0067] Particularly during landing, mooring, or docking maneuvers, such as with aircraft or watercraft, position and orientation data that can be determined as precisely as possible are of great help in enabling more precise and fine-tuned control of the object. This increases operational safety when controlling vehicles such as airplanes, helicopters, drones, cargo ships, or other aircraft or watercraft; and it can also minimize noise emissions and fuel consumption. By simultaneously and, if necessary, redundantly checking the position data from different detection devices, for example, a stationary detection device and a mobile detection device, a plausibility check can be performed when the localization data is combined. This advantageously increases the safety and reliability of the localization and thus also the control of the vehicle.In cases of limited line of sight, for example in bad weather conditions, wavelengths in the non-visible range can also be used, such as infrared light or UV light.

[0068] The concept can be supplemented by implementing special two-dimensional planar patterns and / or shapes of the holographic optical elements, so that the intensity of the reflected ambient light generates color-dependent intensity maxima depending on the distance of the marker from the optoelectronic detection device. Furthermore, special holographic optical deflection elements with retroreflectivity maxima at very high scattering angles relative to the surface normal of the two-dimensional planar marker can be provided, so that the detection of such intensity maxima in the optoelectronic detection device indicates that a critical angular range has been reached in which marker orientation can no longer be guaranteed with sufficient reliability due to the insufficient detectable marker area. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2021 / 0248338 A1

[0003] US 10 885 413 B1

[0003] DE 10 2017 218 544 A1

[0023]

Claims

[1] Localization system (1) for the time-resolved location and / or position determination of an object (100; 200) marked with several optoelectronically detectable markers (20), comprising: an optoelectronic detection device (30) comprising: an optoelectronic sensor (32) which is designed to detect, in a time-resolved manner, light emanating from holographic optical elements (HOE; 21, 22, 23) applied to the markers (20), in particular diffracted and / or reflected and / or backscattered light, and to generate a time-resolved sequence of two-dimensional reflection images; and an image processing device (34) which is coupled to the optoelectronic sensor (32) and is designed to determine a time series of positions and / or orientations of the object relative to the optoelectronic detection device (30) by evaluating the spectrally resolved local intensity distribution in the two-dimensional reflection images detected by the optoelectronic sensor (32). [2] Localization system (1) according to claim 1, wherein the optoelectronic detection device (30) further comprises a memory (33) which is coupled to the image processing device (34) and stores a plurality of comparison and / or training data for the detected two-dimensional reflection images in a database. [3] Localization system (1) according to claim 1 or 2, wherein the image processing device (34) is further designed to carry out an object recognition of structures of a contrast code (10) of the markers (20) recorded in the two-dimensional reflection images captured by the optoelectronic sensor (32). [4] Localization system (1) according to one of claims 1 to 3, wherein at least one of the holographic optical elements (21, 22, 23) applied to the markers (20) has a holographic deflector (24) with high angular selectivity at low elevation angles relative to the surface of the marker (20). [5] Localization system (1) according to one of claims 1 to 4, wherein the color depth of the optoelectronic sensor (32) is at least 8 bits. [6] Localization system (1) according to one of claims 1 to 5, wherein at least one of the holographic optical elements (21, 22, 23) applied to the markers (20) has an optical function, so that a changing distance of the detection device (30) to the object (100; 200) is indicated by a color change and / or a change in the intensity distribution in the two-dimensional reflection images detected by the optoelectronic sensor (32). [7] Localization system (1) according to one of claims 1 to 6, wherein at least one of the holographic optical elements (21, 22, 23) applied to the markers (20) has an optical function, so that a changing object orientation for dedicated angular ranges is indicated by a color change and / or a change in the intensity distribution in the two-dimensional reflection images detected by the optoelectronic sensor (32). [8] Localization system (1) according to one of claims 6 and 7, wherein at least two of the holographic optical elements (21, 22, 23) applied to the markers (20) have duplicated optical functions for improving the reliability of a determination of the distance of the detection device (30) to the object (100; 200) and / or a determination of the object orientation for dedicated angular ranges. [9] Localization method (M) for the time-resolved location and / or position determination of an object (100; 200) marked with several optoelectronically detectable markers (20), comprising the steps: time-resolved detection (M1) of light emanating from holographic optical elements (HOE; 21, 22, 23) applied to the markers (20), in particular diffracted and / or reflected and / or backscattered, by means of an optoelectronic sensor (32) of an optoelectronic detection device (30); generating (M2) a time-resolved sequence of two-dimensional reflection images by the optoelectronic sensor (32) from the detected light; Evaluating (M3) the spectrally resolved local intensity distribution in the two-dimensional reflection images captured by the optoelectronic sensor (32); and Determining (M4) a time series of positions and / or orientations of the object (100; 200) relative to the optoelectronic detection device (30) or relative to surrounding markers (20) on the basis of the evaluated spectrally resolved local intensity distribution. [10] Localization method (M) according to claim 9, wherein the evaluation (M3) of the spectral and / or local intensity distribution in the two-dimensional reflection images acquired by the optoelectronic sensor (32) comprises comparing the two-dimensional reflection images with a plurality of comparison and / or training data stored in a database for the acquired two-dimensional reflection images. [11] Localization method (M) according to one of claims 9 and 10, further comprising the steps: Carrying out (M5) an object recognition of structures of a contrast code (10) of the markers (20) recorded in the two-dimensional reflection images captured by the optoelectronic sensor (32). [12] Localization method (M) according to one of claims 9 to 11, wherein at least one of the holographic optical elements (21, 22, 23) applied to the markers (20) has a holographic deflector (24) with high angular selectivity at low elevation angles relative to the surface of the marker (20). [13] Localization method (M) according to one of claims 9 to 12, wherein the color depth of the optoelectronic sensor (32) is at least 8 bits. [14] Localization method (M) according to one of claims 9 to 13, wherein at least one of the holographic optical elements (21, 22, 23) applied to the markers (20) has an optical function, so that a changing distance of the detection device (30) to the object (100; 200) or surrounding markers (20) is indicated by a change in the spectral and / or local intensity distribution in the two-dimensional reflection images detected by the optoelectronic sensor (32). [15] Localization method (M) according to one of claims 9 to 14, wherein at least one of the holographic optical elements (21, 22, 23) applied to the markers (20) has an optical function, so that a changing object orientation for dedicated angular ranges is indicated by a change in the spectral and / or local intensity distribution in the two-dimensional reflection images detected by the optoelectronic sensor (32). [16] Localization method (M) according to one of claims 14 and 15, wherein at least two of the holographic optical elements (21, 22, 23) applied to the markers (20) have duplicated optical functions for improving the reliability of a determination of the distance of the detection device (30) to the object (100; 200) and / or a determination of the object orientation for dedicated angular ranges. [17] Navigation method for a land, water or air vehicle (100; 200) marked with a plurality of optoelectronically detectable markers (20) and / or an environment marked with detectable markers (20), comprising the steps: time-resolved detection (M1) of light emanating from holographic optical elements (HOE; 21, 22, 23) applied to the markers (20), in particular diffracted and / or reflected and / or backscattered, by means of an optoelectronic sensor (32) of an optoelectronic detection device (30); generating (M2) a time-resolved sequence of two-dimensional reflection images by the optoelectronic sensor (32) from the detected reflected light; Determining (M4) a navigation path of the land, water or air vehicle (100; 200) relative to the optoelectronic detection device (30) or the environment marked with detectable markers (20) on the basis of the time-resolved sequence of two-dimensional reflection images. [18] Navigation method according to claim 17, comprising the step: Generating navigation control information for the land, water or air vehicle (100; 200) for implementing the determined navigation path, wherein the navigation control information is dependent on an optical function of the holographic optical elements (21, 22, 23) applied to at least one of the markers (20).

Images