Thermal image processing methods, computer-readable storage media, and electronic devices

Abstract

The present invention provides a thermal image processing method, a computer-readable storage medium, and an electronic device. The method includes: obtaining initial thermal images and several phase images obtained by thermal imaging module and time-of-flight ranging module respectively capturing images of the same target scene; overlaying several phase images to obtain a reference image; obtaining a depth image based on several phase images; obtaining a target thermal image based on the initial thermal image, the reference image, and the depth image; obtaining a first frequency domain image based on the target thermal image; generating a second frequency domain image based on the reference image; obtaining a fused thermal image based on the first frequency domain image and the second frequency domain image. The thermal image processing method provided in this application can improve temperature measurement accuracy.

Description

[Technical Field]

[0001] This application relates to image processing technology, and more particularly to a thermal image processing method, a computer-readable storage medium, and an electronic device. [Previous Technology]

[0002] Current thermal imaging cameras form thermal images by detecting the energy distribution of long-wave infrared (LWIR) radiation generated by objects in the environment. However, the temperature measurement results of the thermal images formed in related technologies have large errors. [Summary of the Invention]

[0003] In view of the above problems, this application provides a thermal image processing method, a computer-readable storage medium and an electronic device to improve the accuracy of thermal image temperature measurement of target objects.

[0004] The first aspect of this application provides a thermal image processing method, comprising: acquiring an initial thermal image and a complex phase image obtained by a thermal imaging module and a time-of-flight ranging module respectively acquiring images of the same target scene; overlaying the complex phase image to obtain a reference image; acquiring a depth image based on the complex phase image; acquiring a target thermal image based on the initial thermal image, the reference image, and the depth image; acquiring a first frequency domain image based on the target thermal image; generating a second frequency domain image based on the reference image; and acquiring a fused thermal image based on the first frequency domain image and the second frequency domain image.

[0005] A second aspect of this application provides a computer-readable storage medium storing a computer program. When executed by a controller, the computer program implements the thermal image processing method as described in any of the preceding claims.

[0006] A third aspect of this application provides an electronic device, including a thermal imaging module, a time-of-flight ranging module, and a controller. The controller is used to execute the thermal image processing method as described in any of the preceding claims.

[0007] The thermal image processing method provided in this application acquires an initial thermal image and a complex phase image using a thermal imaging module and a time-of-flight ranging module, respectively. Then, a reference image and a depth image are acquired based on the complex phase image. The temperature of the initial thermal image is corrected based on the distance information carried in the depth image and the object category and corresponding emissivity identified in the reference image to obtain a target thermal image. Finally, the two-frequency domain images obtained from the target thermal image and the reference thermal image are fused to obtain a fused thermal image with high temperature accuracy and rich image detail. The thermal image processing method provided in this application can effectively improve the temperature measurement accuracy of thermal images.

Implementation Method

[0008] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0009] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0010] Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0011] Current thermal imaging cameras form thermal images by detecting the energy distribution of long-wave infrared (LWIR) radiation generated by objects in the environment. However, the temperature measurement results of the thermal images formed in related technologies have large errors.

[0012] In view of this, this application provides a thermal image processing method, a computer-readable storage medium, and an electronic device to improve the accuracy of thermal image temperature measurement of target objects.

[0013] Please refer to Figure 1, which is a functional module schematic diagram of an electronic device 100 provided in an embodiment of this application. The image processing method provided in this embodiment is applied to the electronic device 100, specifically, it can be executed by a controller disposed in the electronic device 100. The electronic device 100 includes, but is not limited to, a memory 11, at least one controller 12, and a camera module 14, which are interconnected by a communication bus 13. The camera module 14 can be an imaging device disposed on the electronic device 100, such as a thermal imaging module 141 and a time-of-flight ranging module 142.

[0014] Please refer to Figure 2. The thermal imaging module 141 and the time-of-flight ranging module 142 can be disposed on the same substrate 143, so that the relative position between the thermal imaging module 141 and the time-of-flight ranging module 142 remains unchanged.

[0015] It is understood that Figure 1 is merely an example of electronic device 100 and does not constitute a limitation on electronic device 100. In actual applications, electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or replace different components. For example, electronic device 100 may also include input / output devices, network access devices, and display screens.

[0016] In the embodiments of this application, the electronic device 100 may be a wearable device, such as a smart helmet, smart glasses, etc. This application does not limit the specific form of the extended display device.

[0017] Please continue to refer to Figure 3, which is a flowchart of an image processing method provided in an embodiment of this application. Depending on different needs, the order of the steps in the flowchart may be changed, and some steps may be omitted. Understandably, the steps shown in Figure 3 can be executed by the controller 12 (see Figure 1). Referring to Figure 3, the image processing method may include steps S301-S307.

[0018] Step S301: Obtain the initial thermal image and complex phase image obtained by the thermal imaging module and the time-of-flight ranging module respectively from the same target scene.

[0019] The thermal imaging module 141 is used to detect the infrared energy radiated by an object and output an initial thermal image based on the infrared energy radiated by the object. The time-of-flight ranging module 142 calculates the distance to the object by emitting a light pulse, such as infrared light, and then measuring the time difference from the emission of the light pulse to its reflection back by the object. Specifically, the receiver of the time-of-flight ranging module 142 samples the energy value of the received signal at different phases (such as 0°, 90°, 180°, 270°, etc.). These energy values ​​are used to calculate the phase difference, thereby resolving the distance to the object. The phase difference is proportional to the distance to the object, and the depth value corresponding to each pixel point can be calculated using the phase difference. To improve the accuracy and robustness of the measurement, the time-of-flight ranging module 142 repeats the above process at multiple different time points to generate multiple phase images. Understandably, a phase image is a two-dimensional image composed of the phase values ​​of all pixels. The phase value of each pixel point in the image represents the distance information from that point to the camera.

[0020] In some embodiments, before performing step S301, the thermal imaging module 141 and the time-of-flight ranging module 142 can be calibrated using a calibration pattern board to correct their intrinsic and extrinsic parameters, so that the initial thermal image and phase image obtained by the thermal imaging module 141 and the time-of-flight ranging module 142 from the same target scene are aligned.

[0021] In some embodiments, step S301 is executed only when a synchronization signal is received, thus ensuring that the images acquired by the thermal imaging module 141 and the time-of-flight ranging module 142 remain synchronized. The synchronization signal can be triggered by user operation or output by the controller 12. This application does not limit the source of the synchronization signal.

[0022] Step S302: Overlay complex phase images to obtain a reference image.

[0023] In some embodiments, the same primitive of the complex phase image can be averaged or weighted separately, thereby superimposing the complex phase images to obtain a reference image. This application does not limit the specific algorithm used in step S302.

[0024] Step S303: Obtain the depth image based on the complex phase image.

[0025] In some embodiments, the distance corresponding to each primitive point can be calculated based on the complex phase image, and the distance of each primitive point can be converted into the gray value of the corresponding primitive to generate a depth image. For example, the gray value in the depth image is positively correlated with the distance. Thus, the larger the gray value of a primitive in the depth image, the farther the distance from the corresponding primitive to the time-of-flight ranging module. Conversely, the smaller the gray value of a primitive in the depth image, the closer the distance from the corresponding primitive to the time-of-flight ranging module.

[0026] Step S304: Obtain the target thermal image based on the initial thermal image, the reference image and the depth image.

[0027] Understandably, thermal imaging modules typically preset a single emissivity and correct the measured temperature using a blackbody. However, the target scene may include different objects. Thus, apart from the material corresponding to the single emissivity, other objects in the target scene may cause temperature conversion errors due to the mismatch between the preset emissivity value and the actual value. In step S304, the corresponding object can be identified based on the reference image, and then the corresponding emissivity can be determined according to the object category to correct the temperature of the corresponding object.

[0028] Furthermore, the intensity of long-wave infrared (LWIR) radiation generated by an object's thermal radiation decreases with distance as it propagates through the air. Therefore, when using the thermal imaging module 141 to measure temperature, the measurement error increases with the distance from the object. Thus, in step S304, the temperature in the initial thermal image can be corrected based on the distance in the depth image.

[0029] Thus, in step S304, a more accurate target thermal image can be obtained based on the initial thermal image, the reference image, and the depth image.

[0030] In some embodiments, step S304 includes substeps S401-S404.

[0031] Step S401: Correct the initial thermal image based on the depth image to obtain the corrected thermal image.

[0032] Understandably, the grayscale value of each pixel in the depth image is used to represent the distance of the corresponding pixel from the time-of-flight ranging module 142. Thus, in step S401, the distance of the corresponding pixel can be obtained by acquiring the grayscale value corresponding to each pixel in the depth image.

[0033] In some embodiments, step S401 includes the following sub-steps:

[0034] Step S4011: Obtain the gray value corresponding to each primitive in the depth image.

[0035] Step S4012: Correct the temperature of the corresponding element in the initial thermal image according to the gray value to obtain the corrected thermal image.

[0036] In some embodiments, a lookup table operation can be performed to obtain the corresponding temperature compensation value based on the grayscale value. Then, a corrected temperature value is calculated based on the temperature compensation value and the temperature value of each element in the initial thermal image. The corrected temperature value replaces the temperature value of each element in the initial thermal image to obtain a corrected thermal image. That is, the temperature value included in each element of the corrected thermal image is the temperature value corrected by the depth image.

[0037] In some embodiments, a lookup table mapping grayscale values ​​to temperature compensation values ​​can be pre-established and stored in memory 11. In other embodiments, multiple lookup tables of grayscale values ​​and temperature compensation values ​​can be stored based on factors such as air humidity and density. In this way, more precise temperature correction can be achieved in step S401.

[0038] Step S402: Determine the objects included in the target scene and their corresponding object categories based on the reference image.

[0039] In some embodiments, the reference image may be processed based on the YOLO algorithm, the DETR algorithm, or a neural network to obtain the objects included in the reference image and their corresponding object categories. This application does not limit the object recognition algorithm used in step S402.

[0040] Step S403: Determine the corresponding emissivity based on the object category of each object.

[0041] In some embodiments, a lookup table operation can be performed to obtain the corresponding emissivity based on the object category of each object. Similarly, a lookup table mapping object categories to emissivity can be pre-established and stored in memory 11. In this way, the emissivity corresponding to each object can be determined by executing step S403.

[0042] Step S404: Obtain the target thermal image based on the corrected thermal image and the emissivity corresponding to each object.

[0043] In step S404, since the initial thermal image and the phase image are aligned with each other, the corrected thermal image obtained by processing the initial thermal image is also aligned with the reference image obtained by processing the complex phase image. In this way, the objects identified in step S403 will also appear in the corrected thermal image. Therefore, the temperature error in the corrected thermal image can be further reduced based on the object category and corresponding emissivity identified in step S403.

[0044] In some embodiments, step S404 includes the following sub-steps:

[0045] Step S4041: Obtain the primitive region corresponding to each object in the corrected thermal image.

[0046] In some embodiments, the region of interest corresponding to each object obtained in the calculation process of step S403 can be directly obtained, and then the corresponding primitive region can be obtained in the corrected thermal image based on each region of interest.

[0047] Step S4042: Correct the temperature corresponding to each primitive region according to the primitive region and the corresponding emissivity to obtain the target thermal image.

[0048] In some embodiments, the temperature corresponding to each primitive region can be determined according to the following formula: Tc=Tm×[(1 / ϵO)^(1 / 4)]

[0049] Where Tc represents the temperature of each primitive region after correction, Tm represents the temperature of each primitive region in the corrected thermal image, and ϵO represents the emissivity of the primitive region.

[0050] In some embodiments, the temperature of all elements within a graphic element region can be averaged or weighted to obtain the temperature of the corresponding graphic element region.

[0051] Next, the temperature of the primitive region calculated based on the emissivity is updated to the corrected thermal image to generate the target thermal image.

[0052] Thus, the target thermal image can be obtained by executing step S4042.

[0053] In summary, by executing steps S401 to S404, a target thermal image can be obtained, thus achieving step S304.

[0054] Step S305: Obtain the first frequency domain image based on the target thermal image.

[0055] In some embodiments, the target thermal image may be subjected to Fourier transform to obtain a first frequency domain image.

[0056] Step S306: Generate a second frequency domain image based on the reference image.

[0057] In some embodiments, the reference image may be subjected to Fourier transform to obtain a second frequency domain image.

[0058] In another embodiment, to enrich image details, step S305 may include steps S601-S602.

[0059] Step S601: Perform image enhancement processing on the reference image to generate the target reference image.

[0060] In some embodiments, the reference image may be filtered, edge detected, etc., to generate the target reference image. This application does not limit the algorithm used in step S601.

[0061] Step S602: Perform Fourier transform and high-pass filtering on the target reference image to generate a second frequency domain image.

[0062] Thus, by performing steps S601 to S602, a second frequency domain image can be generated based on the reference image.

[0063] Step S307: Obtain the fused thermal image based on the first frequency domain image and the second frequency domain image.

[0064] In some embodiments, step S307 includes the following sub-steps:

[0065] Step S701: Overlay the first frequency domain image and the second frequency domain image to generate the third frequency domain image.

[0066] In some embodiments, the first frequency domain image and the second frequency domain image can be directly subjected to a weighted average process, assigning different weights to the spectral components of the two images, and then superimposing them. In other embodiments, both the first frequency domain image and the second frequency domain image can be decomposed into sub-images of different frequency bands, and then fused in each frequency band. For example, a larger weight is used in the low frequency band, while a smaller weight is used in the high frequency band. This application does not limit the specific algorithm used in step S701.

[0067] Step S702: Perform inverse Fourier transform on the third frequency domain image to obtain the fused thermal image.

[0068] In summary, the image processing method provided in this application acquires an initial thermal image and a complex phase image using a thermal imaging module and a time-of-flight ranging module, respectively. Then, a reference image and a depth image are acquired based on the complex phase image. Furthermore, the temperature of the initial thermal image is corrected based on the distance information carried in the depth image and the object category and corresponding emissivity identified in the reference image to obtain a target thermal image. Finally, the two frequency domain images obtained from the target thermal image and the reference thermal image are fused to obtain a fused thermal image with high temperature accuracy and rich image details. The thermal image processing method provided in this application can effectively improve the temperature measurement accuracy of thermal images.

[0069] Please refer to Figures 4 and 5 together. Figure 4 is a flowchart of the image change process involved in the thermal image processing method in an embodiment of this application. Figure 5 is a schematic diagram of the process involved in executing the thermal image processing method provided in this application. Understandably, the thermal imaging module 141 outputs an initial thermal image a, and the time-of-flight ranging module 142 outputs a complex phase image b. Then, the complex phase image b is superimposed to obtain a reference image b1, and a depth image b2 is obtained based on the complex phase image b. Next, the initial thermal image a is corrected based on the depth image b2 to obtain a corrected thermal image a1. At the same time, the objects included in the reference image b1 and their corresponding object categories, such as a coffee machine and a human, are determined based on the reference image b1. Furthermore, the corresponding emissivity is determined based on the object category of each object, and then the target thermal image a2 is obtained based on the corrected thermal image a1 and the emissivity corresponding to each object. Then, a Fourier transform is performed on the target thermal image a2 to obtain the first frequency domain image a3, and image enhancement, Fourier transform, and high-pass filtering are performed on the reference image b1 to obtain the second frequency domain image b3. Finally, the first frequency domain image a3 and the second frequency domain image b3 are fused to obtain the fused thermal image c.

[0070] Thus, after the above thermal image processing, the temperature of the coffee machine nozzle is corrected from 26° in the initial thermal image a to 73° in the fused thermal image c, and the human body temperature is corrected from 32° in the initial thermal image a to 36° in the fused thermal image c, which greatly improves the accuracy of the temperature displayed by the thermal image.

[0071] Referring to Figures 1 and 2 together, this application also provides an electronic device 100, including a thermal imaging module 141, a time-of-flight ranging module 142, and a controller 12. The controller 12 is used to execute the thermal image processing method provided in any of the above embodiments.

[0072] It is understood that the controller 12 mentioned above can be a central processing unit (CPU), or other general-purpose controllers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose controller can be a microcontroller or controller 12, or any conventional controller, etc. Controller 12 is the control center of electronic device 100, connecting various parts of the entire electronic device through various interfaces and lines.

[0073] The electronic device 100 also includes a memory 11. The memory 11 can be used to store computer programs and / or modules / units. The controller 12 implements various functions of the electronic device 100 by running or executing the computer programs and / or modules / units stored in the memory 11 and by calling the data stored in the memory 11. The memory 11 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created according to the use of the electronic device (such as video data, audio data, phone book, etc.). In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk drives, memory, plug-in hard disk drives, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk memory device, flash memory device, or other volatile solid-state memory devices.

[0074] Referring to FIG6, another embodiment of this application also provides a computer-readable storage medium 200, which stores a computer program 210 including at least one instruction, the at least one instruction being executed by a controller in an electronic device to implement the thermal image processing method as described in any of the above embodiments.

[0075] The exemplary computer program 210 may be divided into one or more modules / units, one or more of which are stored in a computer-readable storage medium 200 and executed by the controller 12 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, the instruction segments being used to describe the execution process of the computer program in an electronic device.

[0076] The present application implements all or part of the processes in the methods of the above embodiments. A computer program that instructs related hardware can be stored in a computer-readable storage medium. When executed by a controller, the computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, an executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the electronic device embodiments described above are merely illustrative. For example, the division of modules is merely a logical functional division, and there may be other division methods in actual implementation.

[0078] In addition, the functional modules in the various embodiments of this application can be integrated into the same processing module, or each module can exist physically separately, or two or more modules can be integrated into the same module. The integrated module can be implemented in hardware form or in the form of hardware plus software functional modules.

[0079] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of this application is defined by the appended claims rather than the foregoing description, and therefore all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is apparent that the word "comprising" does not exclude other modules or steps, and the singular does not exclude the plural. Multiple modules or electronic devices stated in the electronic device claim may also be implemented by the same module or electronic device by means of software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0080] Furthermore, the above figures are merely illustrative descriptions of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting.

[0081] Furthermore, the above figures are merely illustrative descriptions of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting.

[0082] This application is not limited to the specific embodiments described above. The scope of protection of this application shall be determined by the content of the patent application. [Simplified Explanation of the Diagram]

[0083] Figure 1 is a structural block diagram of an electronic device provided in an embodiment of this application.

[0084] Figure 2 is a schematic diagram of the structure of a camera module provided in an embodiment of this application.

[0085] Figure 3 is a schematic flowchart of a thermal image processing method provided in an embodiment of this application.

[0086] Figure 4 is a flowchart of the image change process involved in the thermal image processing method in one embodiment of this application.

[0087] Figure 5 is a schematic diagram of image processing involved in a thermal image processing method provided in an embodiment of this application.

[0088] Figure 6 is a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. [Biomaterial Storage]

[0090] None

Claims

1. A thermal image processing method, improved in that it includes: The process involves acquiring an initial thermal image and a complex phase image from the same target scene using a thermal imaging module and a time-of-flight ranging module, respectively; overlaying the complex phase image to obtain a reference image; acquiring a depth image based on the complex phase image; acquiring a target thermal image based on the initial thermal image, the reference image, and the depth image; acquiring a first frequency domain image based on the target thermal image; generating a second frequency domain image based on the reference image; and acquiring a fused thermal image based on the first frequency domain image and the second frequency domain image.

2. The thermal image processing method as described in claim 1, wherein, The step of obtaining a target thermal image based on the initial thermal image, the reference image, and the depth image includes: correcting the initial thermal image based on the depth image to obtain a corrected thermal image; determining the objects and corresponding object categories included in the target scene based on the reference image; determining the corresponding emissivity based on the object category of each object; and obtaining the target thermal image based on the corrected thermal image and the emissivity corresponding to each object.

3. The thermal image processing method as described in claim 2, wherein, The step of correcting the initial thermal image based on the depth image to obtain a corrected thermal image includes: obtaining the grayscale value corresponding to each pixel in the depth image; and correcting the temperature of the corresponding pixel in the initial thermal image based on the grayscale value to obtain the corrected thermal image.

4. The thermal image processing method as described in claim 2, wherein, Determining the corresponding emissivity based on the object category of each object includes: obtaining the corresponding emissivity by looking up a table based on the object category of each object.

5. The thermal image processing method as described in claim 2, wherein, The step of obtaining the target thermal image based on the corrected thermal image and the emissivity corresponding to each of the objects includes: obtaining a primitive region in the corrected thermal image corresponding to each of the objects; and correcting the temperature corresponding to each primitive region based on the primitive region and the corresponding emissivity to obtain the target thermal image.

6. The thermal image processing method as described in claim 1, wherein, The step of generating a second frequency domain image based on the reference image includes: performing image enhancement processing on the reference image to generate a target reference image; and performing Fourier transform and high-pass filtering on the target reference image to generate the second frequency domain image.

7. The thermal image processing method as described in claim 1, wherein, The step of obtaining the fused thermal image based on the first frequency domain image and the second frequency domain image includes: superimposing the first frequency domain image and the second frequency domain image to generate a third frequency domain image; and performing an inverse Fourier transform on the third frequency domain image to obtain the fused thermal image.

8. The thermal image processing method as described in claim 1, wherein, Before acquiring the initial thermal image and complex phase image obtained by the thermal imaging module and the time-of-flight ranging module from the same target scene respectively, the method further includes: when a synchronization signal is received, performing the step of acquiring the initial thermal image and complex phase image obtained by the thermal imaging module and the time-of-flight ranging module from the same target scene respectively.

9. A computer-readable storage medium storing a computer program, wherein the improvement is that, when the computer program is executed by a controller, it implements the thermal image processing method as described in any one of claims 1 to 8.

10. An electronic device, improved in that the electronic device includes a thermal imaging module, a time-of-flight ranging module, and a controller, the controller being configured to perform the thermal image processing method as described in any one of claims 1 to 8.

Images