Thermal image processing method, computer readable storage medium, and electronic device
Patent Information
- Application Number
- CN202510172123.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-17
- Publication Date
- 2026-08-18
AI Technical Summary
然而，相关技术中形成的热图像的测温结果误差较大
[0007] The thermal image processing method provided in this application acquires an initial thermal image and several phase images using a thermal imaging module and a time-of-flight ranging module, respectively. Then, a reference image and a depth image are obtained based on the phase images. The temperature of the initial thermal image is corrected based on the distance information carried in the depth image and the object category and corresponding emissivity identified in the reference image to obtain a target thermal image. Finally, the two frequency domain images obtained from the target thermal image and the reference thermal image are fused, resulting in a fused thermal image with high temperature data accuracy and rich image details. The thermal image processing method provided in this application can effectively improve the temperature measurement accuracy of thermal images.
Smart Images
Figure CN122591066A_ABST
Abstract
Description
Technical Field
[0001] This application relates to image processing technology, and more particularly to a thermal image processing method, a computer-readable storage medium, and an electronic device. Background Technology
[0002] Current thermal imaging cameras generate thermal images by detecting the energy distribution of long-wave infrared (LWIR) radiation produced by objects in the environment. However, the temperature measurement results of the thermal images generated by these technologies have relatively large errors. Summary of the Invention
[0003] In view of this, this application provides a thermal image processing method, a computer-readable storage medium, and an electronic device to improve the accuracy of thermal image temperature measurement of target objects.
[0004] A first aspect of this application provides a thermal image processing method, comprising: acquiring an initial thermal image and several phase images obtained by a thermal imaging module and a time-of-flight ranging module respectively acquiring images of the same target scene; overlaying the several phase images to obtain a reference image; acquiring a depth image based on the several phase images; acquiring a target thermal image based on the initial thermal image, the reference image, and the depth image; acquiring a first frequency domain image based on the target thermal image; generating a second frequency domain image based on the reference image; and acquiring a fused thermal image based on the first frequency domain image and the second frequency domain image.
[0005] A second aspect of this application provides a computer-readable storage medium storing a computer program. When executed by a controller, the computer program implements the thermal image processing method as described in any of the preceding claims.
[0006] A third aspect of this application provides an electronic device including a thermal imaging module, a time-of-flight ranging module, and a controller. The controller is used to execute the thermal image processing method as described in any of the preceding claims.
[0007] The thermal image processing method provided in this application acquires an initial thermal image and several phase images using a thermal imaging module and a time-of-flight ranging module, respectively. Then, a reference image and a depth image are obtained based on the phase images. The temperature of the initial thermal image is corrected based on the distance information carried in the depth image and the object category and corresponding emissivity identified in the reference image to obtain a target thermal image. Finally, the two frequency domain images obtained from the target thermal image and the reference thermal image are fused, resulting in a fused thermal image with high temperature data accuracy and rich image details. The thermal image processing method provided in this application can effectively improve the temperature measurement accuracy of thermal images. Attached Figure Description
[0008] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be considered as a limitation on the scope of protection of this application. In the various drawings, similar components are numbered similarly.
[0009] Figure 1 This is a structural block diagram of an electronic device provided in an embodiment of this application.
[0010] Figure 2 This is a schematic diagram of the structure of a camera module provided in one embodiment of this application.
[0011] Figure 3 This is a schematic flowchart of a thermal image processing method provided in an embodiment of this application.
[0012] Figure 4 This is a flowchart illustrating the image change process involved in a thermal image processing method according to an embodiment of this application.
[0013] Figure 5 This is a schematic diagram of image processing involved in a thermal image processing method provided in an embodiment of this application.
[0014] Figure 6 This is a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation
[0015] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.
[0016] It should be noted that when a component is considered to be "connected" to another component, it can be directly connected to the other component or may also have an intervening component. When a component is considered to be "placed" on another component, it can be directly placed on the other component or may also have an intervening component. The terms "top," "bottom," "upper," "lower," "left," "right," "front," "back," and similar expressions used in this article are for illustrative purposes only.
[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.
[0018] Some embodiments will now be described with reference to the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0019] Current thermal imaging cameras generate thermal images by detecting the energy distribution of long-wave infrared (LWIR) radiation produced by objects in the environment. However, the temperature measurement results of the thermal images generated by these technologies have relatively large errors.
[0020] In view of this, this application provides a thermal image processing method, a computer-readable storage medium, and an electronic device to improve the accuracy of thermal image temperature measurement of target objects.
[0021] Please see Figure 1 , Figure 1 This is a functional block diagram of an electronic device 100 provided in an embodiment of this application. The image processing method provided in this embodiment is applied to the electronic device 100, specifically, it can be executed by a controller disposed in the electronic device 100. The electronic device 100 includes, but is not limited to, a memory 11, at least one controller 12, and a camera module 14, which are interconnected via a communication bus 13. The camera module 14 can be an imaging device disposed on the electronic device 100, such as a thermal imaging module 141 and a time-of-flight ranging module 142.
[0022] Please see Figure 2 The thermal imaging module 141 and the time-of-flight ranging module 142 can be disposed on the same substrate 143, so that the relative position between the thermal imaging module 141 and the time-of-flight ranging module 142 remains unchanged.
[0023] Understandably, Figure 1 This is merely an example of electronic device 100 and does not constitute a limitation on electronic device 100. In actual applications, electronic device 100 may include more or fewer components than shown, or combine certain components, or replace different components. For example, electronic device 100 may also include input / output devices, network access devices, and displays.
[0024] In the embodiments of this application, the electronic device 100 may be a wearable device, such as a smart helmet, smart glasses, etc. This application does not limit the specific form of the extended display device.
[0025] Please continue reading. Figure 3 , Figure 3 This is a flowchart of the image processing method provided in one embodiment of this application. Depending on different needs, the order of the steps in the flowchart can be changed, and some steps can be omitted. Understandably, Figure 3 The steps shown can be controlled by controller 12 (refer to...) Figure 1 ) Execution. See also Figure 3 As shown, the image processing method may include the following steps: Step S301: Obtain the initial thermal image and several phase images obtained by the thermal imaging module and the time-of-flight ranging module from the same target scene.
[0026] The thermal imaging module 141 detects the infrared energy radiated by an object and outputs an initial thermal image based on this energy. The time-of-flight ranging module 142 calculates the distance to the object by emitting a light pulse, such as infrared light, and measuring the time difference between its emission and reflection. Specifically, the receiver of the time-of-flight ranging module 142 samples the energy value of the received signal at different phases (e.g., 0°, 90°, 180°, 270°, etc.). These energy values are used to calculate the phase difference, thereby resolving the object's distance. The phase difference is proportional to the object's distance, and the depth value corresponding to each pixel can be calculated using this phase difference. To improve measurement accuracy and robustness, the time-of-flight ranging module 142 repeats the above process at multiple different time points, generating multiple phase images. Understandably, a phase image is a two-dimensional image composed of the phase values of all pixels. The phase value of each pixel in the image represents the distance information from that point to the camera.
[0027] In some embodiments, before performing step S301, the thermal imaging module 141 and the time-of-flight ranging module 142 can be calibrated using a calibration pattern board to correct their intrinsic and extrinsic parameters, thereby aligning the initial thermal image and phase image obtained by the thermal imaging module 141 and the time-of-flight ranging module 142 from image acquisition of the same target scene.
[0028] In some embodiments, step S301 is executed only when a synchronization signal is received, thereby ensuring that the images acquired by the thermal imaging module 141 and the time-of-flight ranging module 142 remain synchronized. The synchronization signal can be triggered by user operation or output by the controller 12. This application does not limit the source of the synchronization signal.
[0029] Step S302: Overlay several phase images to obtain a reference image.
[0030] In some embodiments, the same pixel of several phase images can be averaged or weighted separately, thereby superimposing the several phase images to obtain a reference image. This application does not limit the specific algorithm used in step S302.
[0031] Step S303: Obtain a depth image based on several phase images.
[0032] In some embodiments, the distance corresponding to each pixel can be calculated based on several phase images, and the distance of each pixel can be converted into the grayscale value of the corresponding pixel to generate a depth image. For example, the grayscale value in the depth image is positively correlated with the distance. Thus, the larger the grayscale value of a pixel in the depth image, the farther the corresponding pixel is from the time-of-flight ranging module; conversely, the smaller the grayscale value of a pixel in the depth image, the closer the corresponding pixel is to the time-of-flight ranging module.
[0033] Step S304: Obtain the target thermal image based on the initial thermal image, reference image, and depth image.
[0034] Understandably, thermal imaging modules typically preset a single emissivity and correct the measured temperature using a blackbody. However, the target scene may include different objects. Thus, besides the material corresponding to that single emissivity, other objects in the target scene will cause temperature conversion errors because the default emissivity value does not match the actual value. In step S304, the corresponding object can be identified based on the reference image, and then the corresponding emissivity can be determined according to the object category to correct the temperature of the corresponding object.
[0035] Furthermore, the intensity of long-wave infrared (LWIR) radiation generated by an object's thermal radiation decreases with distance as it propagates through the air. Therefore, when using the thermal imaging module 141 to measure temperature, the measurement error increases with the distance from the object. Thus, in step S304, the temperature in the initial thermal image can be corrected based on the distance in the depth image.
[0036] Thus, in step S304, a more accurate target thermal image can be obtained based on the initial thermal image, the reference image, and the depth image.
[0037] In some embodiments, step S304 includes the following sub-steps: Step S401: Correct the initial thermal image based on the depth image to obtain a corrected thermal image.
[0038] Understandably, the grayscale value of each pixel in the depth image is used to represent the distance of the corresponding pixel from the time-of-flight ranging module 142. Thus, in step S401, the distance of the corresponding pixel can be obtained by acquiring the grayscale value of each pixel in the depth image.
[0039] In some embodiments, step S401 includes the following sub-steps: Step S4011: Obtain the grayscale value corresponding to each pixel in the depth image.
[0040] Step S4012: Correct the temperature of the corresponding pixel in the initial thermal image based on the grayscale value to obtain the corrected thermal image.
[0041] In some embodiments, a lookup table operation can be performed to obtain the corresponding temperature compensation value based on the grayscale value. Then, a corrected temperature value is calculated based on the temperature compensation value and the temperature value of each pixel in the initial thermal image. The corrected temperature value replaces the temperature value of each pixel in the initial thermal image to obtain a corrected thermal image. In other words, the temperature value included in each pixel of the corrected thermal image is the temperature value corrected by the depth image.
[0042] In some embodiments, a lookup table mapping grayscale values to temperature compensation values can be pre-established and stored in memory 11. In other embodiments, multiple lookup tables of grayscale values and temperature compensation values can be stored based on factors such as air humidity and density. This allows for more precise temperature correction in step S401.
[0043] Step S402: Determine the objects included in the target scene and their corresponding object categories based on the reference image.
[0044] In some embodiments, image processing of the reference image can be performed based on the YOLO algorithm, the DETR algorithm, or a neural network to obtain the objects included in the reference image and their corresponding object categories. This application does not limit the object recognition algorithm used in step S402.
[0045] Step S403: Determine the corresponding emissivity based on the object category of each object.
[0046] In some embodiments, a lookup table operation can be performed to obtain the corresponding emissivity based on the object category of each object. Similarly, a lookup table mapping object categories to emissivity can be pre-established and stored in memory 11. Thus, the emissivity corresponding to each object can be determined by executing step S403.
[0047] Step S404: Obtain the target thermal image based on the corrected thermal image and the emissivity corresponding to each object.
[0048] In step S404, since the initial thermal image and the phase image are aligned, the corrected thermal image obtained by processing the initial thermal image is also aligned with the reference image obtained from several phase images. As a result, the objects identified in step S403 will also appear in the corrected thermal image. Therefore, based on the object category and corresponding emissivity identified in step S403, the temperature error in the corrected thermal image can be further reduced.
[0049] In some embodiments, step S404 includes the following sub-steps: Step S4041: Obtain the pixel region corresponding to each object in the corrected thermal image.
[0050] In some embodiments, the region of interest corresponding to each object obtained in step S403 can be directly obtained, and then the corresponding pixel region can be obtained in the corrected thermal image based on each region of interest.
[0051] Step S4042: Correct the temperature of each pixel region according to the pixel region and the corresponding emissivity to obtain the target thermal image.
[0052] In some embodiments, the temperature corresponding to each pixel region can be determined according to the following formula: Where Tc represents the temperature of each pixel region after correction, and Tm represents the temperature of each pixel region in the corrected thermal image. This represents the emissivity of the pixel region.
[0053] In some embodiments, the temperatures of all pixels within a pixel region can be averaged or weighted to obtain the temperature of the corresponding pixel region.
[0054] Next, the temperature of the pixel region calculated based on emissivity is updated to the corrected thermal image to generate the target thermal image.
[0055] Thus, a target thermal image can be obtained by executing step S4042.
[0056] In summary, by executing steps S401 to S404, a target thermal image can be obtained, thus achieving step S304.
[0057] Step S305: Obtain the first frequency domain image based on the target thermal image.
[0058] In some embodiments, the target thermal image may be subjected to a Fourier transform to obtain a first frequency domain image.
[0059] Step S306: Generate a second frequency domain image based on the reference image.
[0060] In some embodiments, a Fourier transform can be performed on the reference image to obtain a second frequency domain image.
[0061] In another embodiment, to enrich image details, step S305 may include: Step S601: Perform image enhancement processing on the reference image to generate the target reference image.
[0062] In some embodiments, the reference image may be filtered, edge detected, etc., to generate the target reference image. This application does not limit the algorithm used in step S601.
[0063] Step S602: Perform Fourier transform and high-pass filtering on the target reference image to generate a second frequency domain image.
[0064] Thus, by executing steps S601 to S602, a second frequency domain image can be generated based on the reference image.
[0065] Step S307: Obtain the fused thermal image based on the first frequency domain image and the second frequency domain image.
[0066] In some embodiments, step S307 includes the following sub-steps: Step S701: Overlay the first frequency domain image and the second frequency domain image to generate the third frequency domain image.
[0067] In some embodiments, the first and second frequency domain images can be directly weighted and averaged, assigning different weights to the spectral components of the two images, and then superimposed. In other embodiments, both the first and second frequency domain images can be decomposed into sub-images of different frequency bands, and then fused within each band. For example, a larger weight can be used in the low-frequency band, while a smaller weight can be used in the high-frequency band. This application does not limit the specific algorithm used in step S701.
[0068] Step S702: Perform an inverse Fourier transform on the third frequency domain image to obtain a fused thermal image.
[0069] In summary, the image processing method provided in this application acquires an initial thermal image and several phase images using a thermal imaging module and a time-of-flight ranging module, respectively. Then, a reference image and a depth image are obtained based on the phase images. Furthermore, the temperature of the initial thermal image is corrected based on the distance information carried in the depth image and the object category and corresponding emissivity identified in the reference image to obtain a target thermal image. Finally, the two frequency domain images obtained from the target thermal image and the reference thermal image are fused, resulting in a fused thermal image with high temperature data accuracy and rich image details. The thermal image processing method provided in this application can effectively improve the temperature measurement accuracy of thermal images.
[0070] Please refer to the following: Figure 4 and Figure 5 , Figure 4 This is a flowchart illustrating the image change process involved in a thermal image processing method according to an embodiment of this application. Figure 5This is a schematic flowchart illustrating the process of executing the thermal image processing method provided in this application. Understandably, the thermal imaging module 141 outputs an initial thermal image a, and the time-of-flight ranging module 142 outputs several phase images b. Then, the several phase images b are superimposed to obtain a reference image b1, and a depth image b2 is obtained based on the several phase images b. Next, the initial thermal image a is corrected based on the depth image b2 to obtain a corrected thermal image a1. Simultaneously, the objects included in the reference image b1 and their corresponding object categories, such as a coffee machine and a human, are determined based on the reference image b1. Furthermore, the corresponding emissivity is determined based on the object category of each object, and then a target thermal image a2 is obtained based on the corrected thermal image a1 and the emissivity corresponding to each object. Then, a Fourier transform is performed on the target thermal image a2 to obtain a first frequency domain image a3, and image enhancement processing, Fourier transform, and high-pass filtering are performed on the reference image b1 to obtain a second frequency domain image b3. Finally, the first frequency domain image a3 and the second frequency domain image b3 are fused to obtain a fused thermal image c.
[0071] Thus, through the above thermal image processing, the temperature of the coffee machine nozzle is corrected from 26° in the initial thermal image a to 73° in the fused thermal image c, and the human body temperature is corrected from 32° in the initial thermal image a to 36° in the fused thermal image c, greatly improving the accuracy of the temperature displayed by the thermal image.
[0072] Please refer to the following: Figure 1 and Figure 2 This application also provides an electronic device 100, including a thermal imaging module 141, a time-of-flight ranging module 142, and a controller 12. The controller 12 is used to execute the thermal image processing method provided in any of the above embodiments.
[0073] It is understood that the controller 12 mentioned above can be a central processing unit (CPU), or other general-purpose controllers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose controller can be a microcontroller, controller 12, or any conventional controller. Controller 12 is the control center of the extended reality device 100, connecting various parts of the electronic device through various interfaces and lines.
[0074] The memory 11 can be used to store computer programs and / or modules / units. The controller 12 implements various functions of the extended reality device 100 by running or executing the computer programs and / or modules / units stored in the memory 11 and calling the data stored in the memory 11. The memory 11 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created according to the use of the electronic device (such as video data, audio data, telephone book, etc.), etc. In addition, the memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0075] Please see Figure 6 Another embodiment of this application provides a computer-readable storage medium 200 storing a computer program 210 including at least one instruction, which is executed by a controller in an electronic device to implement the thermal image processing method as described in any of the above embodiments.
[0076] For example, computer program 210 may be divided into one or more modules / units, one or more of which are stored in computer-readable storage medium 200 and executed by controller 12 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing specific functions, the instruction segments describing the execution process of the computer program in an electronic device.
[0077] This application implements all or part of the processes in the methods of the above embodiments. Alternatively, a computer program can instruct related hardware to implement these processes. The computer program can be stored in a computer-readable storage medium. When executed by a controller, the computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, an executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.
[0078] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the electronic device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0079] Furthermore, the functional modules in the various embodiments of this application can be integrated into the same processing module, or each module can exist physically separately, or two or more modules can be integrated into the same module. The integrated modules described above can be implemented in hardware or in the form of hardware plus software functional modules.
[0080] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that it can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other modules or steps, and the singular does not exclude the plural. Multiple modules or electronic devices recited in the electronic device claims may also be implemented by the same module or electronic device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.
[0081] Furthermore, the above figures are merely illustrative descriptions of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting.
[0082] This application is not limited to the specific embodiments described above. Those skilled in the art will readily understand that many alternative solutions exist for the test fixture without departing from the principles and scope of this application. The scope of protection of this application is determined by the claims.
Claims
1. A thermal image processing method, characterized in that, The method includes: Acquire the initial thermal image and several phase images obtained by the thermal imaging module and the time-of-flight ranging module from the same target scene; A reference image is obtained by overlaying the aforementioned phase images; A depth image is obtained based on the aforementioned phase images; The target thermal image is obtained based on the initial thermal image, the reference image, and the depth image; A first frequency domain image is obtained based on the target thermal image; A second frequency domain image is generated based on the reference image; A fused thermal image is obtained based on the first frequency domain image and the second frequency domain image.
2. The thermal image processing method according to claim 1, characterized in that, The step of obtaining the target thermal image based on the initial thermal image, the reference image, and the depth image includes: The initial thermal image is corrected based on the depth image to obtain a corrected thermal image; The objects and their corresponding object categories in the target scene are determined based on the reference image; The corresponding emissivity is determined based on the object category of each of the aforementioned objects; The target thermal image is obtained based on the corrected thermal image and the emissivity corresponding to each of the objects.
3. The thermal image processing method according to claim 2, characterized in that, The step of correcting the initial thermal image based on the depth image to obtain a corrected thermal image includes: Obtain the grayscale value corresponding to each pixel in the depth image; The temperature of the corresponding pixel in the initial thermal image is corrected based on the grayscale value to obtain the corrected thermal image.
4. The thermal image processing method according to claim 2, characterized in that, The step of determining the corresponding emissivity based on the object category of each of the objects includes: The corresponding emissivity is obtained by looking up a table based on the object category of each object.
5. The thermal image processing method according to claim 2, characterized in that, The step of obtaining the target thermal image based on the corrected thermal image and the emissivity corresponding to each object includes: Obtain the pixel region corresponding to each of the objects in the corrected thermal image; The temperature of each pixel region is corrected according to the pixel region and the corresponding emissivity to obtain the target thermal image.
6. The thermal image processing method according to claim 1, characterized in that, The step of generating a second frequency domain image based on the reference image includes: The reference image is subjected to image enhancement processing to generate a target reference image; The target reference image is subjected to Fourier transform and high-pass filtering to generate the second frequency domain image.
7. The thermal image processing method according to claim 1, characterized in that, The step of obtaining the fused thermal image based on the first frequency domain image and the second frequency domain image includes: The first frequency domain image and the second frequency domain image are superimposed to generate a third frequency domain image; Perform an inverse Fourier transform on the third frequency domain image to obtain the fused thermal image.
8. The thermal image processing method according to claim 1, characterized in that, Before acquiring the initial thermal image and several phase images obtained by the thermal imaging module and the time-of-flight ranging module from the same target scene, the method further includes: When a synchronization signal is received, the steps of acquiring the initial thermal image and several phase images obtained by the thermal imaging module and the time-of-flight ranging module from the same target scene are executed.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the controller, it implements the thermal image processing method as described in any one of claims 1 to 8.
10. An electronic device, characterized in that, The electronic device includes a thermal imaging module, a time-of-flight ranging module, and a controller, wherein the controller is used to execute the thermal image processing method as described in any one of claims 1 to 8.