How to generate a thermal image
A single IR sensor system with positioning and multiple sweep directions generates high-resolution thermal images, addressing the cost and bulkiness issues of traditional cameras, enabling low-cost thermal imaging devices.
Patent Information
- Application Number
- JP2021577248
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-07-05
- Filing Date
- 2020-07-01
- Publication Date
- 2024-11-07
- Estimated Expiration
- 2040-07-01
AI Technical Summary
Thermal imaging cameras are expensive and bulky due to the use of multiple sensors, necessitating a cost-effective alternative.
A single IR sensor system with positioning means to determine its position and orientation, combined with multiple sweep directions, is used to generate thermal images by calculating temperature values from overlapping and non-overlapping fields of view, allowing for low-cost thermal imaging devices like mobile phones.
This approach provides high-resolution thermal imaging at a lower cost by using a single IR sensor, enabling real-time feedback and image improvement through user-guided sweeping, effectively replacing bulky multi-pixel thermal imaging chipsets.
Smart Images

Figure 0007579286000001 
Figure 0007579286000002 
Figure 0007579286000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a system and method for collecting and processing information from an IR sensor (infrared sensor) to generate a thermal image, and to a method for generating thermal imaging data. [Background technology]
[0002] Thermographic cameras are known. Most thermographic cameras use lenses to focus IR radiation onto a chip containing multiple sensors. Such sensors are expensive and bulky. There is a need for a simple, low-cost thermographic camera. Summary of the Invention
[0003] In a first aspect of the invention, there is provided a method for providing thermal image data using an apparatus comprising an IR sensor configured to receive IR radiation from its surroundings within a field of view and position determination means capable of determining a position and an orientation of said IR sensor, said method comprising the steps of: a) the IR sensor determines, at least at a predetermined frequency, an IR sensor value corresponding to a temperature; b) Sweeping the field of view over the area of interest from which the thermal image data is (to be) generated. c) for each determination in step a) while the field of view is being swept, said position determining means determines a current relative position and orientation of each of said devices. d) using the determined IR sensor values together with their respective detected positions and orientations to determine thermal imaging data of said region of interest. Here, a temperature for an area in the image is determined by determining a temperature for two overlapping fields of view such that an overlap between the two fields of view is associated with a non-common field of view for the two overlapping fields of view, and a temperature value for the non-common field of view is determined by using a temperature difference between the two overlapping fields of view and a ratio of the area of the non-common field of view to the area of the fields of view.
[0004] In particular, there is provided a method for providing thermal image data using an apparatus comprising an IR sensor configured to receive IR radiation from the surroundings within a field of view and position determination means capable of determining the position and orientation of said IR sensor, said thermal image data representing a number of picture elements (pixels) in a dot matrix or bitmap, said method comprising the steps of: a) the IR sensor determines, at least at a predetermined frequency, an IR sensor value corresponding to a temperature; b) sweeping a field of view over the area of interest from which the thermal imaging data is generated. c) for each determination in step a) while the field of view is being swept, said position determining means determines a current relative position and orientation of each of said devices. d) using the determined IR sensor values together with their respective detected positions and orientations to determine a thermal image of said area. Here, a temperature for a pixel in the image is determined by determining a temperature for two overlapping fields of view such that an overlap between the two fields of view is associated with a non-common field of view for the two overlapping fields of view, and a temperature value for the non-common field of view is determined by using a temperature difference between the two overlapping fields of view and a ratio of the area of the non-common field of view to the area of the fields of view, and the temperature value thus determined is used to determine a temperature for a pixel in the thermal imaging data. The single sensor design provides a cost-effective way of providing thermal imaging in low-cost devices that have a single IR sensor, such as a cell phone.
[0005] In one embodiment, the entire temperature difference between the two partially overlapping fields of view is assumed to be caused by one or both of the non-common fields of view. In one embodiment, the entire temperature difference between the two partially overlapping fields of view is assumed to be caused by one of the non-common fields of view.
[0006] The fields of view may be swept in at least two different sweep directions, resulting in a first temperature value for the non-common fields of view from the first sweep direction and a second temperature value for the non-common fields of view from the second sweep direction, and the first and second temperature values are used to determine a temperature for an area within the image or pixel representing an area shared by both of the two non-common fields of view. Using multiple sweep directions improves the resolution of the image. Adding more sweep directions improves the resolution.
[0007] The data from the various sweep directions can be used in any suitable manner to determine the temperature of the shared region, in one embodiment, the temperature for a region or pixel in the image is determined by averaging the temperatures of the shared regions.
[0008] The IR sensor may be provided in a handheld device, for example a mobile device such as a mobile phone. The user may be provided with instructions to sweep the device in at least two different directions that are not in the detection direction of the field of view of the IR sensor. The instructions to the user may include instructions to move the handheld device in a predetermined direction, the certain direction being updated according to data from the position and orientation determining means. If the device has a display, the display may be used to display instructions prompting the user to sweep and may also be used to display a thermal image generated using the thermal imaging data. The display may be used to display the image in real time as the image is formed. This has the advantage of allowing real time feedback to the user on how to improve the image.
[0009] In a second aspect of the present invention, there is provided a system comprising an IR sensor configured to receive IR radiation from the surroundings within a field of view and a position determination means capable of determining a position and orientation of the IR sensor, the system configured to provide thermal imaging data, the system configured to cause the IR sensor to determine an IR sensor value corresponding to a temperature at least at a predetermined frequency, and further configured to determine a current relative position and orientation of the IR sensor at the time the IR sensor value is determined, and further configured to use the determined IR sensor values together with their respective detected positions and orientations to determine data for a thermal image, wherein a temperature for an area in the image is determined by determining a temperature for two overlapping fields of view such that an overlap between the two fields of view is associated with a non-common field of view for the two overlapping fields of view, and the temperature value for the non-common field of view is determined by using the temperature difference between the two overlapping fields of view and the ratio of the area of the non-common field of view to the area of the field of view.
[0010] In particular, a system is provided comprising an IR sensor configured to receive IR radiation from the surroundings within a field of view and position determination means capable of determining a position and orientation of said IR sensor, said system configured to provide thermal image data representative of a plurality of pixels in a dot matrix or bitmap, said system configured to cause said IR sensor to determine an IR sensor value corresponding to a temperature at least at a predetermined frequency, further configured to determine a current relative position and orientation of each of the IR sensors at the time the IR sensor value is determined, and further configured to use the determined IR sensor values together with their respective detected positions and orientations to determine data of the thermal image, wherein the temperature for a pixel in the image is determined by determining a temperature for two partially overlapping fields of view such that an overlap between the two fields of view is associated with a non-common field of view for the two partially overlapping fields of view, and the temperature value for the non-common field of view is determined by using the temperature difference between the two partially overlapping fields of view and the ratio of the area of the non-common field of view to the area of the field of view, and the temperature for said point in the image is determined using the temperature value thus determined for the non-common field of view to which the point belongs.
[0011] In a third aspect of the invention there is provided software configured to repeatedly obtain IR sensor values from an IR sensor and further configured to repeatedly obtain a position and orientation of said IR sensor from a position determining means, said software being further configured to use said IR sensor values together with their respective detected positions and orientations to determine a thermal image of said area, wherein a temperature for the area in the image is determined by determining a temperature for two overlapping fields of view such that an overlap between the two fields of view is associated with a non-common field of view for the two overlapping fields of view, and a temperature value for the non-common field of view is determined by using the temperature difference between the two overlapping fields of view and the ratio of the area of the non-common field of view to the area of the fields of view, and the temperature value thus determined is used to determine a temperature for the area in the thermal imaging data.
[0012] In particular, there is provided software configured to repeatedly obtain IR sensor values from an IR sensor and further configured to repeatedly obtain a position and orientation of said IR sensor from a position determining means, said software further configured to use said IR sensor values together with their respective detected positions and orientations to determine a thermal image of said area, wherein the thermal image data represents a plurality of pixels in a dot matrix or bitmap, and wherein the temperature for a pixel in said image is determined by determining a temperature for two partially overlapping fields of view such that an overlap between the two fields of view is associated with a non-common field of view for the two partially overlapping fields of view, and the temperature value for the non-common field of view is determined by using the temperature difference between the two partially overlapping fields of view and the ratio of the area of the non-common field of view to the area of the fields of view, and wherein the temperature value thus determined is used to determine the temperature for a pixel in the thermal image data. [Brief description of the drawings]
[0013] [Figure 1] FIG. 1 is a schematic diagram of the system. [Diagram 2] FIG. 2 is a schematic diagram of the apparatus. [Diagram 3] FIG. 3 is a schematic diagram of a thermal image. [Figure 4] FIG. 4 is a schematic image of the device and the field of view. [Diagram 5] FIG. 5 is a schematic image of the device and the field of view. [Figure 6] FIG. 6 is a schematic image of the device and the field of view. [Figure 7] FIG. 7 is a schematic image of the device and the field of view. [Figure 8] FIG. 8 is a flow chart illustrating the method. [Figure 9] FIG. 9 is a schematic image of the field of view. [Figure 10] FIG. 10 is a schematic image of the field of view. [Figure 11]FIG. 11 is a schematic image of the field of view. [Figure 12] FIG. 12 is a flow chart illustrating the method. [Figure 13] FIG. 13 is a schematic diagram of the field of view and region of interest. [Figure 14] FIG. 14 is a schematic diagram of the field of view and region of interest. [Figure 15] FIG. 15 is a schematic diagram of a device with a display. [Figure 16] FIG. 16 is a schematic diagram of a device with a display. [Figure 17] FIG. 17 shows the experimental data. [Figure 18] FIG. 18 shows the experimental data. [Figure 19] FIG. 19 shows the experimental data. [Figure 20] FIG. 20 shows the experimental data. [Figure 21] FIG. 21 shows the experimental data. [Figure 22] FIG. 22 shows the experimental data. [Figure 23] FIG. 23 shows the experimental data. [Figure 24] FIG. 24 shows the experimental data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] With reference to Figures 1 to 6, the system 1 comprises an IR sensor 2 arranged to receive IR radiation in a field of view 4. The field of view 4 has a detection direction 5. The IR sensor 2 can be any useful type of IR sensor, for example a thermopile or a bolometer, where a thermopile is preferred due to its short response time. The IR sensor is preferably of a type that provides an absolute temperature value, such as a thermopile. The angle α of the field of view 4 may be between 5° and 120°, more preferably between 6° and 45°, even more preferably between 7° and 15°. The field of view 4 may have any suitable shape, but is preferably circular, as shown in Figures 3, 5 to 7, 9 to 11.
[0015] The IR sensor 2 is preferably of a type that is sensitive to one value, and only one value, at one given moment. The thermal imaging data is generated by scanning the field of view 4 of the IR sensor 2 over an area of interest 3, repeatedly collecting IR information during the scan, and then compiling and processing the collected information with the position and / or orientation of the IR sensor 2 to generate the image data.
[0016] System 1 also includes a memory 8, a processor 9, and a communication bus 10. In a preferred embodiment, system 1 includes a digital display 11 for displaying digital images. Digital display 11 may be, for example, an LCD display. System 1 may include any suitable combination of software and hardware. System 1 may include, for example, an operating system and a device for providing input, such as a touch display.
[0017] The memory 8 is used to store, for example, software for executing the method as well as necessary data such as IR sensor values, positions, orientations, time points, time stamps, overlapping fields of view and their positions, image data, etc.
[0018] The device 1 also comprises a position and / or orientation determining means 6, referred to herein as "position determining means 6", capable of determining the position and / or the orientation of the IR sensor 2. The orientation of the IR sensor 2 means the orientation of the sensing direction 5. Preferably, both the position and the orientation are determined by the position determining means 6, but in certain embodiments only the position or the orientation is determined. The position and the orientation are preferably determined with a fine resolution so that movements of a few millimeters (in the case of translation) or degrees (in the case of orientation) can be detected.
[0019] The position and orientation are determined in a manner known in the art. Thus, the position determining means 6 may comprise any suitable technology, including tracking cameras, marker-based or markerless tracking, inertial tracking, digital model building, gyroscopes, and accelerometers, and these technologies may be combined in any suitable manner. Current models of mobile phones, such as the iPhone X and Android, are capable of determining position and orientation at high resolution. Apple iOS provides an API called Core Motion for providing position and orientation data, and Android provides a similar API.
[0020] Preferably, the IR sensor 2 and the position determining means 6 are included in a device 7, which is a device that may preferably be handheld. The device 7 may also be a mobile phone, such as an iPhone or an Android phone. Nevertheless, parts of the system 1 may be located external to the device 7, such as on a server. In particular, the memory 8 and the processor 9, or parts of the memory 8 and the processor 9, may be located on a server that is in digital communication, e.g. wireless communication, with the device 7.
[0021] The detection direction 5 of the field of view 4 is preferably fixed relative to the device 7. The position determination means 6 are able to determine the detection direction 5 of the field of view 4 and the position of the infrared sensor 2. The position and orientation 5 are preferably determined with respect to surrounding objects, such as objects in the region of interest 3. The position and orientation may preferably be determined with respect to stationary objects. The detection direction 5 may be the same as a line of observation perpendicular to the surface of the display 11 of the device 7 (see figures 5 and 6). A user holding the device 7 with the display 11 in his hand will then observe the display 11 in the same direction as the detection direction 5 of the IR sensor 2, which is advantageous when displaying thermal images in real time (see below).
[0022] System 1 may also include suitable components for processing the signal from IR sensor 2 as known in the art, such as, for example, amplifiers, filters, A / D converters, etc. System 1 is powered by a suitable power source, such as, for example, a battery. An IR signal is provided to system 1 from IR sensor 2. System 1 may use any suitable combination of software and hardware to process the signal, store data representative of the signal, and perform calculations using this data.
[0023] The system 1 is used to generate thermal imaging data in the form of a dot matrix 12 representing a thermal image, an example of which is shown in FIG. 3. The dot matrix 12 comprises picture elements (pixels) 13 which make up the image. The dot matrix image may be a bitmap image. The image data is suitable for rendering the thermal image 12 on a display, for example a display 11 of the system 1. The image data may be arranged as a dot matrix 12 with each pixel 13 storing a temperature value. The number of pixels in the thermal image 12 may be, for example, between 10,000 and several million.
[0024] Each pixel 13 in the dot matrix 12 can take on at least two different colors (e.g. black and white), but preferably can show more colors. Each color represents a temperature value or a range of temperature values. The color scheme is selected depending on the application. Typically a color scale (black blue green yellow red) is used in thermal images. Black represents the lowest temperature and red represents the highest temperature. Each color represents a certain temperature range. The colors are pseudocolors and any useful color scheme can be used. The temperature range for each color is selected based on the intended application. The display 11 of the device 7 may be used to render a thermal image based on the dot matrix 12. The pixels 13 can thus display colors representing temperature values.
[0025] To generate image data, the field of view 4 is swept over the region of interest 3, i.e. the area for which a thermal image 12 is to be generated. The direction of the sweep is indicated by arrow 15 in the figure. If the device 7 is handheld, the sweep is performed manually by moving the hand. However, in some embodiments, a powered and / or automatically controlled sweep can be used, for example, where a motor controlled by a computer moves the field of view 4. Thus, the system 1 and methods described herein can be used as a replacement for a multi-pixel thermal imaging chipset.
[0026] The IR sensor 2 determines IR values within a field of view 4 at least at a given frequency. For a mobile device, a suitable frequency may be between 1 Hz and 100 Hz, where 2 Hz to 20 Hz is more preferred.
[0027] The position determination means 6 repeatedly determines the position and / or orientation of the IR sensor 2 at suitable time intervals. This time interval may be simultaneous or the same frequency as the capture of the IR data. However, a different sampling frequency may be used. The sampling frequency for the position and / or orientation may be the frequency for sampling the IR data. A time pairing algorithm may be used to pair the correct IR data with the correct data for the position and / or orientation. For example, a timestamp may be used. Thus, the position determination means 6 detects, for each IR value while the field of view 4 is swept, the current relative position and / or orientation of each of the IR sensors 2.
[0028] Sweeping the direction of the field of view 4 may be done, for example, by translating (translating) the device 7 so that the field of view 4 moves across the region of interest 3 (translation direction is shown by arrow T in FIG. 5), or by changing the detection direction 5, for example by rotating the device 7 as shown in FIG. 6 (i.e. rotating the device 7 around an axis that does not correspond to the detection direction 5, rotation direction is shown by arrow R), or a combination of both. In a preferred embodiment, both translation and change of direction are used.
[0029] Thus, each IR value detected during the scan is associated with data representing a position and / or orientation. The IR image data may be compiled by using the temperature values and arranging them in a dot matrix 12 together with the use of position and / or orientation data. The positions and / or orientations obtained by the position determining means 6 may be used to position the various captured fields of view in three-dimensional space. As shown in FIG. 3, the field of view 4 (or a portion of the field of view, see below) may assign temperature values to a number of pixels 13. Thus, a pixel 13 in the image 12 typically corresponds to an area smaller than the field of view 4. Two or more pixels 13 may be used to represent an area for which a temperature value has been determined, and one pixel may be used to represent two or more areas for which a temperature has been determined. Sampling may be used to map the various regions where the temperature has been determined to pixels 13 of the image 12. For example, one or more pixels 13 in the image 12 may be used to depict the shared region 21 (see below). Any suitable bitmap digital image file format may be used, such as JPEG, TIFF, GIF, etc. Vector graphics may also be used, but it may be easier to use a bitmap / dot matrix file format.
[0030] A preferred embodiment is shown in Figures 7-9, where the sampling and sweeping rates are such that there is a partial overlap between the fields of view 4a, 4b, and the two fields of view 4a, 4b are associated with two fields of view 102, 103 that are not common to the two overlapping fields of view 4a, 4b.
[0031] In FIG. 7, the direction of the sweep is again indicated by an arrow 15. The "old" non-common field of view 102 is the part of the field of view 4a captured before the field of view 4b. The "new" non-common field of view 103 is the part of the field of view 4b captured after the field of view 4a. In various embodiments, different assumptions are made as to what influences the detected change in temperature between the regions 4a and 4b. The temperature difference between the old field of view 4a and the new field of view 4b can be due to the old non-common region 102 or the new non-common region 103, or both. The temperature value for the at least one non-common field of view 102 103 is determined by using the temperature difference between the two partially overlapping fields of view 4a 4b and the ratio of the area of the non-common field of view 102 103 to the area of the field of view 4a 4b. The temperature for a pixel 13 in the image 12 is determined by using the temperature value thus determined for the at least one non-common field of view 102 103 to which the pixel 13 belongs.
[0032] The ratio of the area of the non-common parts 102-103 to the field of view 4a-4b depends on the sampling frequency and the speed at which the sweep is performed. The sweep speed may vary, especially when a portable device 7 is used. The ratio of the area of the non-common field of view 102-103 to the area of the field of view 4 may be determined by an area ratio determination means. The area ratio determination means may use the current sweep speed determined by the position determination means 6. The area ratio determination means is part of the system 1 and is preferably implemented as software using information from the position determination means 6 and the sampling time point. Thus, the area ratio determination means may use information from the position determination means 6 to determine the speed at which the field of view 4 moves over the area of interest 3 and use that information to determine the ratio between the areas of the non-common fields of view 102-103 to the fields of view 4a-4b. A fast sweep speed results in a larger non-common area 102-103, whereas a slow sweep speed results in a smaller non-common area. A slow sweep speed improves the resolution of the thermal information. A suitable ratio of non-common fields of view to the total field of view 4 may be between 1 and 20%.
[0033] In one embodiment, the sampling frequency of the IR sensor and the position determining means 6 is automatically increased as the sweep speed increases.
[0034] The change in temperature between the fields of view 4a and 4b is divided by the area ratio, and the resulting ratio is used to calculate the temperature of the old and / or new non-common field of view 102-103. In one embodiment, the entire temperature difference between the overlapping fields of view 4a, 4b is assumed to be due to the new non-common field of view 103. In one embodiment, the entire temperature difference is assumed to be due to the old non-common field of view 102. In a further embodiment, the temperature difference is assumed to be due to both the new non-common field of view 103 and the old non-common field of view 102, e.g., in a fixed ratio, e.g., 50% due to the new non-common field of view 102 and 50% due to the old non-common field of view 103.
[0035] In the following example, we assume that the entire change in temperature from field of view 4a to field of view 4b is due to the new non-common field of view 103. For example, suppose field temperature 4a is 23° C. and the temperature in area 4b is 21° C. The temperature difference (−2° C.) is due to the non-common field of view 102-103. Assuming that non-common field of view 103 has a size that is 15% of field of view 4b, there is a change of -2° C. / 0.15=-13.3° C., i.e., the new non-common field of view 103 temperature value is 23-13.3=9.7° C. As explained below, using data from more sweep directions improves accuracy.
[0036] The method is shown in Figure 8. In step 200, IR sensor data from two overlapping fields of view 4a 4b from the same sweep direction is collected. In step 201, the ratio of the area of the non-common fields of view 102 and 103 to the area of the fields of view 4a 4b is determined by an area ratio determining means. Note that the ratio between the non-common area 102 and the field of view 4a is the same as the ratio of the non-common area 103 to the field of view 4b. Steps 200 and 201 can be performed in any order. In step 202, the temperature of the non-common fields of view 102 and / or 103 is determined.
[0037] The method is preferably applied to a series of overlapping fields of view 4a, 4b, 4c, 4d, etc., such that the method utilizes an overlap between 4a and 4b, an overlap between 4b and 4c, an overlap between 4c and 4d, etc., as shown in FIG. 9. In this way, a substantial part of the area of interest 3 may be covered by non-common fields of view. A starting point 14 may be used to associate the positions and orientations of the different fields of view 4a, b and their non-common fields of view 102, 103 with each other. The starting point 14 is preferably assigned to the first sampled field of view 4a. The positioning means 6 can build a model of how the fields of view 4 move and map the IR sensor values against the starting point 14 to generate the thermal image 12. The starting point 14 is preferably a point in a space defined relative to the surroundings or the area of interest 3. The starting point 14 is preferably associated with an initial direction of observation 5.
[0038] A larger region of interest 3 can be covered by sweeping it multiple times with an appropriate offset each time (FIG. 10). The offset should preferably be at most the diameter of the field of view 4.
[0039] 11-12, the field of view 4 is preferably swept over the region of interest 3 in at least two different directions 15a,b. For example, a point 20 is covered by a new non-common field of view 103a from a first direction 15a and a new non-common field of view 103b from a second direction 15b. The temperature of the point 20 or region 21 is determined using the temperature values of the first and new second non-common fields of view 103a,b as described above, and then the temperature values of the point 20 or region 21 from the at least two sweep directions 15a,15b are used to determine a temperature value for a region in the thermographic image 12, such as pixel 13 or a number of pixels in the thermographic image 12.
[0040] In step 300, the field of view is swept over the region of interest 3 in a first direction 15a, resulting in a first temperature value for a first new non-common field of view 103a. This is done using the method described above with reference to Figures 7-10. In step 301, the field of view 4 is swept over the region of interest 3 in a second direction 15b, similarly resulting in a second temperature for a second new old non-common field of view 103b. In step 303, it is determined that the first and second old non-overlapping fields of view 103a,b cover more or less the same area, shown as a shared area 21. The first and second temperature values are used to determine a temperature for the shared area 21, which may be used to determine temperature values for pixels 13 within the shared area 21 of the image 12. The shared area 21 (dashed line) has the same temperature value.
[0041] The shared region 21 is determined using the stored location information for each non-common region 103a, 103b. For example, a start location 14 can be used and all temperature values can be spatially associated with the start location 14.
[0042] Although the above describes how new non-common regions 103a,b are used, the old non-common regions 102a,b could be used instead, or both the old and new non-common regions could be used as described above, where any temperature change is split in a predetermined manner, for example 50 / 50, between the old and new regions.
[0043] In a preferred embodiment, values from three, four, five, six, seven, eight or more sweep directions 15 are used. Thus, temperature values from three, four, five, six, seven, eight or more non-overlapping regions 102, 103 can be used. Using temperature data from multiple sweep directions (such as two or more) increases the resolution of the thermal image 12. Temperature data from multiple sweep directions can be used in any suitable manner. In one embodiment, an average of temperature values from different directions 15ab is used. In one embodiment with three or more sweep directions, the value furthest from the average is used. In one embodiment, the extreme values closest to the average are used (if the extreme values are 10 and 2 and the average is 5, then 2 is selected). In one embodiment, the median value is used.
[0044] Thus, for each pixel 13 in the image 12, the temperature value may be the result of using temperature data for multiple temperature values, for example an average of multiple temperature values. It is preferred that the sweep directions 15a, 15b of the two data sets are separated by at least 45°. Preferably, the two sweep directions are separated by 90°. In a preferred embodiment, the field of view 4 is swept in at least four mutually different directions 15, resulting in four different sets of determined IR sensor values. The mutually different directions 15 are preferably 90° apart. In an even more preferred embodiment, eight different directions 15 separated by 45° are used. In one embodiment, three different sweep directions 15 separated by 120° are used.
[0045] The sweeps are preferably performed in one or the same parallel plane, which is preferably perpendicular to the observation direction 5 of the field of view 4 at the starting point 14.
[0046] 13, the field of view 4 is swept several times in a first direction 15a with a vertical offset such that each sweep covers at least a partially different area, and the field of view 4 is swept additionally several times in a second direction 15b with a different vertical offset such that each sweep 16 covers at least a partially different area of the region of interest 3. Each arrow 16 indicates the path that the center of gravity of the projection of the field of view 4 takes to cover the region of interest 3.
[0047] However, random sweeps may also be used, as shown in FIG. 14. For example, a user holding the device 7 in his hand may sweep the field of view 4 over the area of interest 3 in a cross-like manner to gather information from several directions. In one embodiment, the image is displayed in real time on the display 11, and the user is given feedback as the image improves during the sweep. Thus, the user can create a thermal image with a "painting"-like motion. This is particularly useful when the image 12 is displayed in real time on the display 11 (see below). The user can improve the image 12 as required by sweeping across the area of interest 3 that needs improvement, and see the results in real time.
[0048] The user may be provided with instructions to sweep in different predetermined directions as described above, for example, two directions 90° apart. The instructions may be provided in any manner, such as text, symbols, or audio, such as synthetic voice. In a preferred embodiment, the instructions are provided on the display 11. An example is shown in FIG. 15. For example, a portable device 7 is used by a user, who is prompted to move the field of view 4, for example, by moving the device 7 in different directions 15a, 15b. In a preferred embodiment, the device 7 has a display 11, which is used to prompt the user. The display 11 can be used to indicate to the user the direction 15 in which the user should sweep the device 7. The display 11 can also provide guidance to orient the device 7, and thus the observation direction 5, in the correct direction. As described above, the prompts may be provided at the same time as the real-time image is displayed.
[0049] In one embodiment shown in FIG. 16, instructions to the user are provided in real time using information from the position determination means 6. In particular, the user may be prompted to translate or rotate the device 7 by translating or rotating the field of view 4 in order to maintain a predefined sweep direction and orientation. In FIG. 14, the display 11 shows a spirit level tool consisting of an arrow 17, a fixed cross 18 and an orientation-dependent cross 19 that moves on the display 11 in accordance with the orientation of the device 7 determined by the orientation determination means 6. The user is prompted to keep the orientation-dependent cross 19 aligned with the fixed cross 18. The movement of the arrow 17 can prompt the user to scan in different directions. The arrow 17 may thus be controlled by the position determination means 6 and a predefined pattern for the movement, for example the pattern of FIG. 13. For example, the user may be prompted to sweep first in a first direction 15a with an offset and then in a second direction 15b with the direction separated by an angle as described above. The position and orientation at the starting point 14 may be used to provide guidance to the user, where the user is prompted to hold the device 7 in the same orientation or position as the starting point 14.
[0050] The user may also be prompted to sweep in a plane, which may be a plane perpendicular to the viewing direction 5 at the starting point 14, or which may be the plane of the display 11 of the device 7 at the starting point 14. Commands such as "closer" or "farther" may be used on the display 11 to sweep while keeping the device 7 in a plane.
[0051] The user may also be prompted to sweep at a certain speed. As noted above, a slower sweep speed provides better resolution. Thus, the user may be prompted to possibly increase or decrease the sweep speed. Feedback may be provided in real time.
[0052] In one embodiment, the display 11 of the device 7 is used to display a thermal image 12 using the data relating to the thermal image processed as described above. The method may thus include the step of displaying the thermal image 12 on the display 11 of the device 7 including the IR sensor 2. The image 12 may be rendered in real time on the display 11 as a user sweeps across the area of interest 3. Thus, image data may be provided to the display 11 for rendering in real time, thereby providing feedback to the user.
[0053] Imaging software for carrying out the methods described herein is also provided. The imaging software may be provided as an app downloaded to a mobile device 7 having the IR sensor 2 and the position determining means 6. The imaging software is preferably stored in a memory 8. The imaging software may be configured to obtain IR sensor values and position and / or orientation data from the position determining means 6. For example, the imaging software may repeatedly query one or more sensors or switch on one or more sensors and provide a predefined configuration for one or more sensors. Such a configuration may for example be instructions to the IR sensor 2 and / or the position determining means 6 regarding a sampling frequency. The imaging software may be configured to pair the IR sensor data with the position and / or orientation data. The imaging software includes a ratio determining means as described above.
[0054] The imaging software is configured to compile the data to generate thermal imaging data to generate thermal image 12. Thus, the imaging software is configured to determine non-overlapping fields of view, and positions and orientations for these, and to compile temperature data from different directions, as described herein.
[0055] Example 1 The method of the present invention was simulated using digital thermal data of a region of interest showing a human and a dog, shown in the top left panel of Figures 17-24. The data was scanned in eight different directions (indicated by arrows) using a virtual field that scanned the digital image data. The position of the virtual field was controlled by the user using a computer mouse. Overlapping fields were determined and the temperature difference between the two overlapping fields was 100% attributed to the new non-common field. The temperatures for up to eight non-overlapping fields were averaged for each pixel in the image. The top right image shows the result of averaging the data set. It can be seen that the image improves as the sweep direction is increased. The bottom right image shows only the average temperature within the field (without overlap or different directions), which is clearly of poorer resolution.
Claims
1. 1. A method for providing thermal imaging data using an apparatus including an IR sensor configured to receive IR radiation from an environment within a field of view and a position determining means capable of determining a position and orientation of said IR sensor, said method comprising: a) the IR sensor determining, at least at a predetermined frequency, an IR sensor value corresponding to a temperature; b) sweeping a field of view over an area that includes the region of interest; c) for each determination in step a) while the field of view is being swept, said position determining means determines a current relative position and orientation of each of said devices; d) using the determined IR sensor values together with their respective detected positions and orientations to determine thermal imaging data of an area including the area of interest; A method in which a temperature for an area including the area of interest is determined by determining a temperature for two partially overlapping fields of view such that an overlap between the two fields of view is associated with a non-common field of view for the two partially overlapping fields of view, and a temperature value for the non-common field of view is determined by using the temperature difference between the two partially overlapping fields of view and a ratio of the area of the non-common field of view to the area of the fields of view.
2. The method of claim 1 , wherein the entire temperature difference between two partially overlapping fields of view is assumed to be caused by one or both of the non-common fields of view.
3. 3. The method of claim 1 or 2, wherein the entire temperature difference between two partially overlapping fields of view is assumed to be caused by one of the non-common fields of view.
4. 4. A method according to any one of claims 1 to 3, wherein the field of view is swept across an area including the region of interest in at least two different sweep directions, resulting in a first temperature value for the non-common field of view from a first sweep direction and a second temperature value for the non-common field of view from a second sweep direction, and the first temperature value and the second temperature value are used to determine a temperature for an area shared by both of the two non-common fields of view.
5. The method of claim 4 , wherein the temperature for the shared area is determined by averaging the temperatures of the shared area.
6. The method of any one of claims 1 to 5, wherein the IR sensor is comprised in a mobile device.
7. The method of claim 6 , wherein instructions are provided to a user to sweep the device in at least two different directions that are not a detection direction of the field of view of the IR sensor.
8. 8. The method of claim 7, wherein the instructions to the user include instructions to move the device in a predetermined direction, the instructions being updated in response to data from the position and orientation determining means.
9. 9. The method of claim 7, wherein the device has a display, the display being used to display instructions to the user to sweep and to display a thermal image generated using the thermal imaging data.
10. 10. The method of claim 1, wherein the device has a display, the display being used to display a thermal image in real time as the thermal imaging data is used to form the thermal image.
11. 1. A system comprising an IR sensor configured to receive IR radiation from the surroundings within a field of view and position determination means capable of determining a position and orientation of said IR sensor, said system being configured to cause said IR sensor to determine, at least at a predetermined frequency, an IR sensor value corresponding to a temperature, and further configured to determine a current relative position and orientation of said IR sensor at a time when said IR sensor value is determined, and further configured to use said determined IR sensor values together with their respective detected positions and orientations to determine data for a thermal image, The system, wherein the temperature in the thermal image is determined by determining a temperature for two partially overlapping fields of view such that an overlap between the two fields of view is associated with a non-common field of view for the two partially overlapping fields of view, and a temperature value for the non-common field of view is determined by using the temperature difference between the two partially overlapping fields of view and a ratio of the area of the non-common field of view to the area of the fields of view.
12. software configured to repeatedly obtain IR sensor values from an IR sensor and further configured to repeatedly obtain a position and orientation of said IR sensor from a position determining means, said software further configured to use said IR sensor values together with their respective detected positions and orientations to determine a thermal image of a predetermined area; The software determines the temperature in the thermal image by determining a temperature for two overlapping fields of view such that an overlap between the two fields of view is associated with a non-common field of view for the two overlapping fields of view, and a temperature value for the non-common field of view is determined by using the temperature difference between the two overlapping fields of view and a ratio of the area of the non-common field of view to the area of the fields of view, and the temperature value thus determined is used to determine the temperature in the thermal image.
Citation Information
Patent Citations
Temperature measuring method for food, and heating cooker using the method
JP1996015038A
Heating cooking appliance
JP1996145376A
Heating cooker
JP2001065870A
Thermal image sensor and air conditioner
JP2015222260A
Methods to combine radiation-based temperature sensor and inertial sensor and / or camera output in a handheld / mobile device
US20140152772A1
