Method for determining the position and / or orientation of a mobile electronic device using a digital thermal imaging camera

The method automates the determination of mobile device position and orientation using thermal imaging by simulating object surface temperatures and performing a spatial backward cut, addressing inefficiencies in existing manual signaling methods.

DE102016119920B4Active Publication Date: 2024-12-05DEUTSCHE TELEKOM AG
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
DE102016119920
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Filing Date
2016-10-19
Publication Date
2024-12-05
Estimated Expiration
2036-10-19

AI Technical Summary

Technical Problem

Existing methods for determining the position and orientation of mobile electronic devices using thermal imaging cameras are inefficient and require manual signaling on the object, which is time-consuming and labor-intensive.

Method used

A method that utilizes a digital thermal imaging camera to automatically determine the position and orientation by simulating temperature values at known object surface points, generating reference data sets through thermal models, and performing a spatial backward cut using internal and external camera orientations, without manual signaling.

Benefits of technology

Enables fully automated determination of the position and orientation of mobile electronic devices using thermal imaging cameras, reducing computational effort and eliminating the need for manual object marking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Method for determining the position and / or orientation of a mobile electronic device using a digital thermal imaging camera (70) whose internal orientation is known, comprising the following steps: a) Recording, with the mobile digital thermal imaging camera (70), a thermal image of an object, wherein the thermal image is produced in a color representation; b) Providing for each predetermined surface area of ​​a predetermined object a reference data set containing position and color information of the respective predetermined surface area, wherein the color information corresponds to a temperature of the respective predetermined surface area; c) Selecting several reference datasets from the reference datasets provided in step b) depending on the result of a comparison of the color information contained in the reference datasets with areas of the color representation of the thermal image acquired in step a); and d) Performing a spatial backward slice to determine the position and orientation of the mobile digital thermal imaging camera (70) using the known internal orientation of the thermal imaging camera (70) and the position information contained in the selected reference datasets.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a method for determining the position and / or orientation of a mobile electronic device with the aid of a digital thermal imaging camera, which is implemented in particular in the mobile electronic device, which can be a smartphone.

[0002] Smartphones with integrated or attachable thermal imaging cameras are known (see, for example, the websites: www.areamobile.de / news / 36787-cat-s60 and www.heise.de / newsticker / meldung / hands-on).

[0003] A method for determining the position and / or orientation of a mobile electronic device using a digital thermal imaging camera is known from DE 10 2016 109 628 A1, which is considered an older patent application according to Section 3 (2) of the German Patent Act (PatentG).

[0004] From DE 10 2014 217 963 A1 a method for determining the pose of data glasses using passive IR markers is known, wherein the data glasses record the environment using an infrared light camera, recognize at least one predefined pattern in the recordings and, depending on the recognized predefined pattern, recognize the pose of the data glasses, i.e. the orientation and position of the data glasses.

[0005] DE 10 2010 038 472 A1 describes a method for processing at least two thermal images recorded using a thermal imaging camera, wherein one thermal image is considered the original image and the other thermal image is considered the reference image. The thermal images have correspondence regions with at least partially matching image information, wherein at least one image parameter is assigned to the image information. Each of the thermal images is assigned a recording position of the thermal imaging camera, wherein the recording positions are each characterized by at least one recording point, a recording angle, and a recording perspective. A difference image is calculated for the two thermal images, which maps differences between image parameters of the original image and image parameters of the reference image. When calculating the difference image, differences between the recording positions of the thermal imaging camera are compensated.In this process, feature correspondences are determined in the thermal image information, which allows the image information of the reference image to be assigned to corresponding image information of the original image. Based on the assigned image information, a deformation map is created in which each piece of image information of the reference image is assigned a position in the original image. The deformation map then forms the basis for subsequent transformation of the thermal images.

[0006] From DE 10 2012 003 256 A1 it is known to automatically derive a 2D or 3D model from a CAD model of a measurement object in order to facilitate the evaluation of thermographic data and, in general, of spatially resolved measurement results and to determine a transformation rule with which image points assigned to the measurement object can be converted into points of the 2D or 3D model.

[0007] US 2013 / 0243250A1 describes a method for determining location, in which an image of an object area is captured using an image recording device and processed by means of a recognition algorithm in order to recognize features and to derive therefrom a first feature descriptor, which is to be compared with feature descriptors stored in a database, each of which is assigned location information, and in case of a match, to enable access to the location information assigned to the matching feature descriptor stored in the database.

[0008] The present invention is based on the object of providing a further method for determining the position and / or orientation of a mobile electronic device using a digital thermal imaging camera.

[0009] A core idea of ​​the invention can be seen in the provision of preferably a plurality of reference surface areas, the positions of which in the object space are known, with respect to at least one object, wherein each reference surface area is assigned, as a rule, time-dependent position and, as a rule, time-dependent color or temperature information after imaging in the image plane of a digital thermal imaging camera, which information is stored as reference data sets.

[0010] The respective temperature of the surface points of the respective reference surface areas can preferably be predicted using a given thermal model. Such a thermal model can be based on a finite element method for decomposing a real 3D structure into a finite number of elements with simplified physical properties, which is explained, for example, in the manual "HYDR Groundwater / Thermal Conductivity Model" by SOFISTIK AG, Oberschleissheim, 2005 (see www.sofistik.de / fileadmin / FILES / produkte / sta). Using a given temperature-color characteristic curve of the digital thermal imaging camera used, the color values ​​for mapping the simulated surface temperatures into the image plane of the mobile digital thermal imaging camera can then be predicted or calculated.

[0011] The above-mentioned technical problem is solved by the method steps of claim 1.

[0012] Thereafter, a method for determining the position and / or orientation of a mobile electronic device using a digital thermal imaging camera whose internal orientation is known is provided, which method comprises the following steps: a) taking, with the mobile digital thermal imaging camera, a thermal image of an object, whereby the thermal image is generated in a color representation; b) providing a reference data set for predetermined surface areas of a predetermined object, which reference data set contains position and color information of the respective predetermined surface area, wherein the color information corresponds to a temperature of the respective predetermined surface area; c) selecting several reference data sets from the reference data sets provided in step b) depending on the result of a comparison of the colour information contained in the reference data sets with areas of the colour representation of the thermal image recorded in step a); and d) performing a spatial backslice to determine the position and / or orientation of the mobile electronic device using the known internal orientation of the thermal imaging camera and the position information contained in the selected reference data sets.

[0013] Spatial backslicing is a method from photogrammetry, which is explained, for example, in the textbook "Photogrammetry" by K. Schwidefsky and F. Ackermann, BG Teubner Stuttgart, 1976.

[0014] It should be noted that orientation preferably refers to the external and / or internal orientation of the thermal imaging camera assigned to the mobile electronic device. The terms "external and internal orientation" are familiar from photogrammetry. They are explained, for example, in the textbook "Photogrammetrie, Grundlagen, Verfahren, Anwendungen" (Photogrammetry, Fundamentals, Methods, Applications) by K. Schwidedfsky et al., 7th edition of "Grundriß der Photogrammetrie" (Outline of Photogrammetry), 1976, BG Teubner Stuttgart. Image coordinate axis scaling based on the pixel size or the design of a digital area sensor can be included in the internal orientation of the thermal imaging camera.

[0015] Furthermore, it should be noted that a position is preferably described by the coordinates of a given 3D coordinate system.

[0016] It should also be noted that a surface area contains at least one surface point. The smallest possible "area" on the object's surface would be a surface point, so the temperature is determined at this point on the object's surface.

[0017] As a rule, however, a predetermined surface area will contain several points, so that the temperature in this surface area is an average representative temperature, which can then be converted into a corresponding color value.

[0018] From a purely practical point of view, in the image plane of a digital thermal imaging camera, the color is not measured in an image point corresponding to a surface point of an object, but rather a representative color value as an average value, for example in a CMOS image pixel (with a dx and dy extension in the image that corresponds to an area of ​​several points on the object surface).

[0019] Advantageous further training is the subject of the subclaims.

[0020] Using a spatial backslice, multiple beams of rays from a camera image converging in object space can be considered to determine the positions of recorded objects in an object space and, at the same time, the position and / or orientation of the camera or mobile electronic device used. However, this requires the assignment of object points to image points, which in practice is primarily achieved through physical signaling on the object, such as by affixing measurement markers and point number information. This so-called signaling must therefore be performed manually in advance and becomes visible in the camera image when the image is recorded.

[0021] In order to be able to carry out the procedure completely automatically, an alternative solution for signaling relevant points in the object space is provided by generating the signaling virtually rather than manually directly on the object.

[0022] This is achieved by step b) comprising the following steps: b1) calling, by a computing unit, at least one thermal model stored in a storage device for the predetermined object; b2) Using the thermal model retrieved in step b1), calculate surface temperature-dependent color information for a plurality of predetermined surface areas of the predetermined object, the positions of which are known. It should be noted that the calculation according to step b2) is advantageously performed for a time and, if appropriate, for a date at which a thermal image was acquired in step a).

[0023] In other words: The signaling on the object is achieved by simulating temperature values ​​at predetermined, i.e. known, object surface points whose positions are known. These temperature values ​​are assigned unique color values ​​in the thermal image via the temperature-color characteristic curve of the thermal imaging camera. If a temperature value of an object point is clearly numerically assigned to a color value in the image and this color value only exists at one point in the image, then an image ray is present at the time of recording. If this color value is present at several points in the image, the person skilled in the art is familiar with various criteria and algorithms for selecting the best corresponding image point fully automatically. Several of these image rays to the best corresponding image points result in a beam of rays that is bundled at the projection center.For example, if at least five rays are present in a beam bundle pointing to known surface points, the position and / or orientation of the thermal imaging camera can be determined if these rays have a known general position. This procedure can be performed iteratively in the Gauss-Markov model, which is well known. Thus, there is a spatial backslice from the object to the projection center of the thermal imaging camera, which corresponds to a single-image beam adjustment.

[0024] The simulation results can be improved if, in step b2), the color information is calculated as a function of time information, and / or information about the material properties of the surface of the predetermined object, and / or weather information, and / or temperature information of the object space in which the predetermined object is located, and / or information about the external and / or internal orientation of a virtual thermal imaging camera and / or information about the position of a virtual thermal imaging camera.

[0025] Instead of obtaining reference data sets through simulation, they can alternatively be created from previously acquired reference thermal images. These reference thermal images can also be acquired by the digital thermal imaging camera. Step b) can include the following steps: Recording, before carrying out step a), at least one reference thermal image of the predetermined object, wherein the at least one reference thermal image is generated in a color representation; Creating, using the reference thermal image, a reference data set for predetermined surface areas of the predetermined object, each of which contains position and color information of the respective predetermined surface area; and storing the reference data sets.

[0026] An advantageous measure which saves energy for the mobile electronic device provides that the thermal image recorded in step a) is transmitted to a remote computer device which is arranged spatially separate from the mobile electronic device, and that steps b) to d) are carried out in the computer device (140).

[0027] If the digital thermal imaging camera is integrated into the mobile electronic device, which may be a mobile phone, steps b) to d) can also be carried out in the mobile electronic device.

[0028] The computational effort for determining the position and / or orientation of the mobile electronic device can be reduced by selecting a thermal image section from the thermal image recorded in step a), by selecting a number of first reference data sets from the provided reference data sets in step b), and by selecting a plurality of first reference data sets from the first reference data sets selected in step b) in step c) depending on the result of a comparison of the color information contained in the first reference data sets with the selected thermal image section.

[0029] The new method thus combines a solution for the automated signaling of an object with the implementation of a spatial backward cut in order to determine the position and / or orientation of the mobile electronic device or the thermal imaging camera fully automatically.

[0030] The invention is explained in more detail below using an exemplary embodiment in conjunction with the accompanying drawings. They show: Fig. 1 an exemplary communication system comprising a mobile phone with an integrated thermal imaging camera, Fig. 2 a house wall with two windows as a predetermined object in a given object space.

[0031] Fig. 1 shows, as an exemplary application, a section of an exemplary communication system 1, which may include a smartphone 10 as a mobile electronic device, an external computing device 140 in the form of a server, and an external database 180. The smartphone 10, the database 180, and the computing device 140 can communicate via an IP network, for example, the Internet 190.

[0032] The server 140 can have a control unit 150, which is embodied, for example, as a microprocessor or microcontroller. Furthermore, the server 140 can contain a data memory 160 and a program memory 170 in which the instructions for controlling and monitoring the server 140 are stored. As explained in more detail below, the server 140, in cooperation with the smartphone 10, can execute several method steps for determining the position and / or orientation of the smartphone 10.

[0033] The smartphone 10 may include a timer 40 that can indicate the current time and / or date. This allows the time and season of a thermal image to be recorded.

[0034] The smartphone 10 can access the internet 190, for example, via a wireless communication interface 130. The smartphone 10 can be configured to obtain, for example, current or past location-specific weather information via the internet 190, which can be stored, for example, in the database 180. The smartphone 10 can also query information about the current position of the sun and / or the duration and / or intensity of sunshine via the internet.

[0035] In addition, the smartphone 10 can contain a temperature sensor 90, a brightness sensor 91 and a motion sensor 92. Furthermore, the smartphone 10 can have a control unit 30, which can be designed as a microprocessor or microcontroller. In the example shown, the smartphone 10 has an integrated digital thermal imaging camera 70. Alternatively or additionally, a digital thermal imaging camera can be connected externally to the smartphone 10 via a corresponding data interface (not shown). It is known that by means of the digital thermal imaging camera 70, a temperature distribution of a recorded surface of an object can be displayed visually, i.e. in color, in an image plane of the thermal imaging camera 70. The thermal imaging camera 70 preferably operates in the thermal infrared (TIR).

[0036] The thermal imaging camera 70 has a known internal orientation, which preferably results from the known focal length of the lens used, the known distortion parameters of the lens used, the principal point position in the image, the position of the distortion symmetry point relative to the principal point of the image, and, if applicable, the pixel scaling of the image plane sensor. The internal orientation can be stored in the smartphone 10.

[0037] Furthermore, in a manner known per se, these variables are used to define a 2D image coordinate system and a local, thermal imaging camera-fixed 3D coordinate system, the origin of which can be defined as the position of the mobile phone 10. Furthermore, a 3D offset or a 3D vector can be defined between the origin of the thermal imaging camera-fixed 3D coordinate system and a mobile phone-fixed point, e.g., the antenna of a GPS receiver 50, to which the GPS receiver 50 can also refer. The GPS receiver 50 of the smartphone 10 can determine the position or position coordinates of the smartphone 10 in a manner known per se.

[0038] The instructions required for controlling and monitoring smartphone 10 can be stored in a program memory 80. It is conceivable that smartphone 10 knows all the information it needs to determine its external orientation, i.e., the position and orientation angle of the mobile phone-based local 3D coordinate system at the time a thermal image is captured. Furthermore, it is conceivable that a temperature-color value characteristic curve relating to thermal imaging camera 70 is stored in smartphone 10, indicating which temperature value is represented by which color value in the image plane.

[0039] As will be explained in more detail below, a plurality of heat models can be stored in the program memory 80, with each of which the surface temperature of a predetermined object, for example that of a house wall 100 of a building and / or that of the window frames of windows 110, 120, which in Fig. 2 can be predicted at least in part.

[0040] The predetermined objects can be located in one or more predetermined object spaces. A predetermined object space can be an exhibition center, a building, a street, or the like. Object spaces can be stored in a well-structured manner in the database 180 to accelerate access to individual objects or thermal models.

[0041] Each object is preferably assigned an object coordinate system in which the 3D positions of its surface are defined. For some surface points of at least one predetermined object, the parameters and algebraic relationships that allow the temperature of these surface points to be calculated or predicted at a specific time can also be stored in the database 180. The necessary equations of the corresponding thermal models can either be loaded into the program memory 80 via the internet 190 and calculated in the microprocessor 30. Alternatively or additionally, these equations or thermal models can already be present in the program memory 80 as part of the installed program.

[0042] In particular, all known necessary imaging equations of the central projection and the internal orientation of the thermal imaging camera, with which the imaging of a pinhole camera can also be simulated, can be stored in the program memory 80 and / or in the database 180 and / or in the program memory 170 of the server 140.

[0043] An algorithm for checking the correspondence between actually recorded thermal images and actually recorded or simulated reference thermal images can also be stored, for example, in the program memory 80 of the smartphone 10, in the program memory 170 and / or in the database 180.

[0044] Using the equations and data from database 180, and / or the sensor data from sensors 90 and 91 of mobile phone 10, and / or the time information from timer 40, the temperature at surface points with known surface coordinates or in surface regions with known 3D positions can be calculated for the at least one predetermined object. For these surface points or surface regions, the calculated temperature values ​​are thus valid for a specific point in time, whereby this point in time should correspond as accurately as possible to the time of the actual thermal image acquisition.The 3D positions with known temperatures are then mathematically projected onto the thermal imaging camera 70's image plane using the internal and, if applicable, external orientation of the camera. At each point on the image plane onto which an object point is projected, the color value resulting from the characteristic curve to be used to convert temperature into color value is stored, for example, in the data memory 60. The result is reference data sets for predetermined surface areas or surface points of a predetermined object, each containing position and color information. It should be noted that a predetermined surface area can comprise one object point or a predeterminable number of object points.

[0045] If a sufficient match is found between the color representation of a currently recorded thermal image and the color information contained in one or more reference data sets, the position information contained in these reference data sets can be used together with the known internal orientation of the thermal imaging camera 70 in a spatial backward section to determine the position and / or orientation of the smartphone 10 or the thermal imaging camera 70.

[0046] The functionality of the smartphone 10 and, if required, the server 140 and the database 180 are explained in more detail below.

[0047] The position and / or orientation of the smartphone 10 or the thermal imaging camera 70 is now to be determined with the help of the integrated thermal imaging camera 70, the user of which is currently in front of a house wall 100, which is in Fig. 2. Two windows 110 and 120 are set in the house wall 100. The window 110 has, for example, a hexagonal window frame 112 made of wood. A temperature sensor 107 can be arranged at each corner of the window frame 112. The window 120 has, for example, a square window frame 122, which can also be made of wood. A temperature sensor 105 can be arranged at each corner of the window frame 122. The temperature sensors 105 and 107 can each be designed to transmit an individual address, such as an IP address, temperature measurements, and the respective measurement times to the smartphone 10 and / or to the server 140 and / or to the database 180 upon request or automatically, depending on the implementation of the system 1. It should be noted that the positions of the six corners of the window frame 112 and also the four corners of the window frame 122 are known in the object space.Furthermore, assume that the user points the thermal imaging camera 70 straight at the window 110, keeping the thermal imaging camera 70 slightly rotated and tilted. The corresponding numerical projection 200 of the image plane of the digital thermal imaging camera 70 into the object, namely the window 110, is shown in . Fig. 2. The principal point of the image is designated by “H”.

[0048] There are various possibilities for providing a reference data set containing position and color information of the respective predetermined surface area for predetermined surface areas or surface points of a predetermined object, for example the six corners of the window frame 112 and the four corners of the window frame 122.

[0049] An exemplary first approach provides that the smartphone 10 and / or the server 140 are each configured to query temperature measurement values ​​from the temperature sensors 105 and 107 at a specific validity time. This is primarily the time at which the user, for example, takes a thermal image of the window 110 with the thermal imaging camera 70. This image is represented by a plurality of pixels, each of which is assigned a surface temperature-dependent color value.Since the positions of the corners of window frames 112 and 122, and thus essentially the positions of temperature sensors 105 and 107, are known, a reference data set for the validity time can be created and provided in smartphone 10 and / or server 140 for each corner point in response to the received IP addresses and temperature measurements of the respective temperature sensors 105 and 107, wherein each reference data set contains position and color information for the respective predetermined surface area, which in the present example is the corner of one of window frames 112 or 122. The color information is calculated in smartphone 10 and / or server 140 depending on a predetermined temperature-color value characteristic curve.

[0050] However, if the temperature sensors 105 and 107 only send their IP address and the temperature values ​​measured at a valid time to the database 180, the smartphone 10 and / or the server 140 are each able to query the IP addresses of the temperature sensors and the associated temperature measured values ​​there.

[0051] In this way, the smartphone 10 and / or the server 140 are capable of providing a reference data set for predetermined surface areas or points of a predetermined object, each of which contains position and color information of the respective predetermined surface area, wherein the color information corresponds to a temperature of the respective predetermined surface area. In the example explained, for example, one reference data set is provided for each corner point of the two window frames 112 and 122, thus a total of ten reference data sets.

[0052] A second exemplary approach provides that a thermal model is stored in the memory 60 of the smartphone 10, and / or in the memory 160 of the server 140, and / or in the database 180 for each window 110 and 120, or only for each window frame 112 and 122. Now, only the case will be considered in which a suitable thermal model is stored in the memory 60 of the smartphone 10 for each window frame 112 and 122. It should be noted that each thermal model describes, in particular, the geometric surface of an object, for example, the window frames 112 and 122.Each thermal model can, for example, take into account information about the material properties of the object's surface, temperature information of the object space, which is, for example, simulated or measured by the temperature sensor 90 of the smartphone 10 and / or by the temperature sensors 105 and 107, time information, such as the time of recording a thermal image of the object with the digital thermal imaging camera 70, information about the external and / or internal orientation of a virtual thermal imaging camera and / or information about the position of a virtual thermal imaging camera, and weather information, which can, for example, be queried by the smartphone 10 from the database 180. Again, it is assumed that at least the corner points of the window frames 112 and 122 are known.The smartphone 10 is capable of calculating, at a valid point in time, which may be the time the window 120 is captured with the thermal imaging camera 70, surface temperature-dependent color information for a plurality of predetermined surface areas of at least one predetermined object, which may be the corners of the window frames 112 and 122, whose positions are known, using the stored thermal models. For each predetermined surface area, the smartphone 10 then creates a reference data set that contains, in addition to the color information, the known position of the respective predetermined surface area. This approach is suitable for determining the position and / or orientation of the smartphone 10 in real time.

[0053] A third exemplary approach provides that in advance, i.e. before the current recording of a thermal image of the Fig. 2, which is required for determining the position and / or orientation of the smartphone 10, at least one reference thermal image of the predetermined object is recorded, wherein the at least one reference thermal image is generated in a color representation using a predetermined temperature-color value characteristic curve of the thermal imaging camera 70. For example, several reference thermal images of the two window frames 112 and 122 are recorded in advance with the digital thermal imaging camera 70 at different times of day and year. Subsequently, using the reference thermal images, a reference data set is created for predetermined surface areas of the at least one predetermined object, in the present example, these are the corner points of the window frames 112 and 122, which reference data set can contain not only the position and color information of the respective surface area, but also the time of day and year at which the reference thermal image was recorded.These reference data sets are stored in advance in the smartphone 10. It is conceivable that such reference thermal images are taken with another thermal imaging camera, whereby the reference data sets obtained therefrom can then be stored, for example, in the server 140 or in the database 180 for later use.

[0054] Now assume that the user of smartphone 10 is standing in front of house wall 100 and wants to know the position and / or orientation of their smartphone 10. Furthermore, several reference data sets are already stored in smartphone 10, which were previously generated from several reference thermal images. Assume that the reference data sets were generated from reference thermal images taken at different times and seasons.

[0055] In order to be able to determine the position and / or orientation with the aid of the digital thermal imaging camera 70, the user takes, for example, a thermal image of the window 110 with the integrated thermal imaging camera 70 at a validity time t0, as shown in Fig. 2. The recording time t0 and the date of the thermal image recording can be determined, for example, by the timer 40 of the smartphone 10.

[0056] In order to locate suitable reference data sets with reduced computational effort, an initial rough positioning of the smartphone 10 can be performed in advance. For example, the user knows the address of the house with wall 100, which they can enter on the smartphone 10. Alternatively, the smartphone 10 can roughly determine its own position using its GPS receiver 50. Assume that the user has entered the address of the house with wall 100 on the smartphone 10. In response to the address, the time, and the season of the thermal image, the smartphone 10 can select the reference data sets belonging to the window frames 112 and 122. Depending on the implementation, the user can also inform the smartphone 10 that the reference data sets belonging to the right window frame 112 should be provided.

[0057] If, for example, the reference data sets are first generated in real time using stored thermal models, the thermal models that describe window frames 112 and 122 are called up by smartphone 10. As previously described, the user can inform smartphone 10 that only the right window 110 is being viewed. In this case, smartphone 10 will only call up the thermal model that describes window frame 112. Using the thermal model and preferably depending on the actual time of recording a thermal image of window frame 112, current weather information, and material properties of window frame 112, smartphone 10 calculates the temperature or the associated color information for several predetermined surface areas. In the example explained, these are the six corner points of window frame 112.

[0058] The color information contained in the provided reference data sets is then compared with the color information of the pixels of the object captured by the thermal imaging camera 70, namely the window frame 112, in order to locate prominent areas in the captured thermal image whose positions are known. The thermal image is represented by a plurality of pixels, each of which is assigned a surface temperature-dependent color value. If the comparison shows that the color information contained in the reference data sets assigned to the corner points of the window frame 112 matches the color information of the currently captured thermal image in a predetermined manner, these reference data sets are selected for further calculation.

[0059] One possible matching criterion can be the exceeding or falling below of a predefined threshold value, which depends on an applied testing algorithm. For example, a threshold value of 0.9 can be stored in data memory 160. A match value of 1 would indicate that the color information contained in a reference data set completely matches the color information of a pixel in the thermal image captured by smartphone 10.

[0060] The position information contained in the selected reference data sets is then read out and, together with the known internal orientation of the thermal imaging camera 70, is used in the equations of a spatial backward section to determine the position and / or orientation of the thermal imaging camera 70 and thus of the smartphone 10.

[0061] It should be noted at this point that the thermal image currently captured by the thermal imaging camera 70 can be transmitted via the Internet from the smartphone 10 to the server 140, in which previously created reference data sets or thermal models for simulating surface temperature values ​​of the windows 110 and 120 can be stored. The color matching previously explained with regard to the smartphone 10 for selecting suitable reference data sets and the subsequent step of performing a spatial backslicing adjustment step can now be carried out in the server 140. The result, i.e., the position and / or orientation of the thermal imaging camera 70 and thus essentially the position and / or orientation of the smartphone 10, can then in turn be transmitted from the server 140 to the smartphone 10.

[0062] In order to reduce the computing effort, especially for the smartphone 10, at least one thermal image section can be specifically selected in the thermal image currently recorded with the thermal imaging camera 70, which, for example, Fig. 2. Similarly, first reference data sets relating to window frame 112 can be specifically selected from previously stored reference data sets. Subsequently, only a color comparison is performed between the color values ​​of the thermal image section and the previously selected first reference data sets in order to find second reference data sets, the position information of which is used in a spatial backward section to determine the position and / or orientation of thermal imaging camera 70 and thus essentially to determine the position and / or orientation of smartphone 10.

[0063] Thanks to the invention, the position and / or orientation of a mobile electronic device 10 can be determined using a thermal imaging camera 70.

[0064] For this purpose, an object, for example, window frames 112 and 122, is specified in object space with its 3D surface. The color representation of the surface in the thermal image of the digital thermal imaging camera 70 is calculated using a known temperature-color characteristic curve. From this, a ray assignment between the object and image space is generated fully automatically without manual signaling at the object. The ray assignment thus generated is used algebraically in a spatial backslice to iteratively determine the position and / or orientation of the thermal imaging camera 70 or the mobile electronic device 10, for example, in the Gauss-Markov model.

Claims

[1] A method for determining the position and / or orientation of a mobile electronic device using a digital thermal imaging camera (70) whose internal orientation is known, comprising the following steps: a) capturing, with the mobile digital thermal imaging camera (70), a thermal image of an object, wherein the thermal image is generated in a color representation; b) providing a reference data set for predetermined surface areas of a predetermined object, which reference data set contains position and color information of the respective predetermined surface area, wherein the color information corresponds to a temperature of the respective predetermined surface area; c) selecting several reference data sets from the reference data sets provided in step b) depending on the result of a comparison of the colour information contained in the reference data sets with areas of the colour representation of the thermal image recorded in step a); and d) performing a spatial backslice to determine the position and orientation of the mobile digital thermal imaging camera (70) using the known internal orientation of the thermal imaging camera (70) and the position information contained in the selected reference data sets. [2] Method according to claim 1, characterized by that step b) comprises the following steps: b1) calling, from a computing unit (30; 150), at least one thermal model stored in a storage device (60; 160; 180) for the predetermined object; b2) Calculating, using the thermal model called in step b1), a surface temperature-dependent color information for a plurality of predetermined surface areas of the predetermined object, the positions of which are known. [3] Method according to claim 2, characterized by that in step b2) the color information is calculated as a function of time information, and / or information about the material properties of the surface of the predetermined object, and / or weather information, and / or temperature information of the object space in which the predetermined object is located, and / or information about the external and / or internal orientation of a virtual thermal imaging camera and / or information about the position of a virtual thermal imaging camera. [4] Method according to claim 1, characterized by that step b) comprises the following steps: Recording, before carrying out step a), at least one reference thermal image of the predetermined object, wherein the at least one reference thermal image is generated in a color representation; Creating, using the reference thermal image, a reference data set for predetermined surface areas of the predetermined object, which reference data set contains position and color information of the respective predetermined surface area; and Saving the reference records. [5] Method according to one of the preceding claims, characterized by that the thermal image recorded in step a) is transmitted to a remote computer device (140), and that steps b) to d) are carried out in the computer device (140). [6] Method according to one of claims 1 to 4, characterized bythat the digital thermal imaging camera is integrated into the mobile electronic device and that steps b) to d) are carried out in the mobile electronic device. [7] Method according to one of the preceding claims, characterized by that a thermal image section is selected from the thermal image recorded in step a), that in step b) a number of first reference data sets are selected from the provided reference data sets, and that in step c) a plurality of first reference data sets are selected from the first reference data sets selected in step b) depending on the result of a comparison of the color information contained in the first reference data sets with the selected thermal image section.

Citation Information

Patent Citations

  • Method for processing thermal images i.e. digital images, for thermographic examination of objects or scenes to detect alterations of insulating properties in building, involves representing base using deformation card

    DE102010038472A1

  • Method for evaluating thermographic data of spatially resolving measurement result, involves automatically determining transformation rule associated with pixels corresponding to item under test

    DE102012003256A1

  • Pose determination of smart glasses using passive IR markers

    DE102014217963A1

  • Method for determining the position and / or orientation of a mobile electronic device using a digital thermal imaging camera

    DE102016109628A1

  • Location of image capture device and object features in a captured image

    US20130243250A1