All-focus infrared image fusion method and device, electronic equipment and medium
By acquiring temperature frames from multiple infrared images, a fusion baseline map is determined, and a full-focus fusion algorithm and convolutional neural network are used to generate a full-focus fusion map. This solves the problem of depth-of-field limitation in infrared thermal imagers and achieves high-definition and aesthetically pleasing images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNI TREND TECH (CHINA) CO LTD
- Filing Date
- 2025-05-14
- Publication Date
- 2026-08-04
AI Technical Summary
Due to depth-of-field issues, infrared thermal imagers cannot guarantee that every detail in the image is clear when taking pictures.
By acquiring multiple infrared images with different focal points, the distribution characteristics of temperature frames are extracted, a fusion baseline image and multiple focal images are determined, and a full-focus fusion algorithm is used to fuse the images. Convolutional neural networks and the Sobel gradient method are combined for feature extraction and weighted superposition to generate a full-focus fused image.
It breaks through the depth-of-field limitations of traditional infrared thermal imagers, improves image detail and clarity, and makes the image visual effect more beautiful.
Smart Images
Figure CN120495097B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of infrared image processing technology, and in particular to a method, apparatus, electronic device and medium for fusion of fully focused infrared images. Background Technology
[0002] Infrared thermal imagers are widely used in various fields such as industry, medical care, security, and environmental protection. However, when users take pictures with infrared thermal imagers, due to the depth-of-field problem of infrared thermal imagers, it is difficult to ensure that every detail of the image is clear.
[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art.
[0004] Application content
[0005] In view of at least one of the above technical problems, this application provides a method, apparatus, electronic device and medium for totalized infrared image fusion.
[0006] Firstly, a fully focused infrared image fusion method is provided, applied to an infrared thermal imager, the method comprising:
[0007] Multiple infrared images are acquired, each with a different focus and a temperature frame.
[0008] The temperature distribution characteristics are extracted from the temperature frame of each infrared image to determine the fusion reference map and multiple focal images;
[0009] A full-focus fusion algorithm is used to fuse the fusion reference image and multiple focal images to obtain a full-focus fused image.
[0010] This fully focused infrared image fusion method can overcome the limitations of traditional infrared thermal imagers in terms of depth of field, and can also improve the detail and clarity of images, making them more visually appealing.
[0011] In some possible implementations, the infrared thermal imager has an AF motor, which is controlled to acquire multiple infrared images.
[0012] In some possible implementations, the fusion reference image and multiple focal images are determined, including:
[0013] Obtain the horizontal and vertical temperature change rates of each infrared image's temperature frame;
[0014] The temperature gradient value of each infrared image frame is calculated based on the horizontal and vertical temperature change rates.
[0015] Calculate the average gradient value of the temperature frame of each infrared image based on the temperature gradient value of each temperature frame.
[0016] The infrared image of the temperature frame with the largest average gradient magnitude is selected as the fusion reference image;
[0017] Multiple focal images are identified, which are infrared images of temperature frames with successively decreasing average gradient magnitudes.
[0018] In some possible implementations, the temperature gradient value of each infrared image temperature frame is calculated using the following formula:
[0019]
[0020] in, The rate of temperature change in the horizontal direction. This represents the rate of temperature change in the vertical direction.
[0021] In some possible implementations, the average gradient value of the temperature frame for each infrared image is calculated using the following formula:
[0022]
[0023] in, is the temperature gradient value, and N is the number of pixels in the infrared image.
[0024] In some possible implementations, a fully focused fusion graph is obtained, including:
[0025] The pre-trained convolutional neural network ResNet is used to extract features from the fused baseline image and multiple focal images. The input is each infrared thermal image grayscale image (1×H×W). The network uses a 2 to 3 layer convolutional + pooling structure to extract local features and outputs multiple sets of semantic feature tensors F1, F2, ... with a shape of C×H×W, where C is the number of channels, H is the image height, and W is the image width.
[0026] For each extracted feature map, ambiguity analysis is performed, and the corresponding ambiguity score map is generated by combining the Sobel gradient method.
[0027] Based on the ambiguity scoring map, a preliminary fusion mask M(x,y) is constructed, and high weights are assigned to clear regions;
[0028] The fusion baseline image and multiple focal images are weighted and superimposed according to the fusion mask. The fusion result is as follows:
[0029]
[0030] Among them, F i (x,y) represents the feature value of the i-th image at position (x,y), Mi (x,y) represents the pixel-level fusion weights.
[0031] Secondly, a fully focused infrared image fusion device is provided, comprising:
[0032] The image acquisition module is used to acquire multiple infrared images, each with a different focus and a temperature frame.
[0033] The focus filtering module is used to extract the temperature distribution characteristics of the temperature frame of each infrared image and determine the fusion reference map and multiple focus images;
[0034] The fusion processing module is used to fuse the fusion reference image and multiple focal images using a full-focus fusion algorithm to obtain a full-focus fused image.
[0035] In some possible implementations, the fusion reference image and multiple focal images are determined, including:
[0036] Obtain the horizontal and vertical temperature change rates of each infrared image's temperature frame;
[0037] The temperature gradient value of each infrared image frame is calculated based on the horizontal and vertical temperature change rates.
[0038] Calculate the average gradient value of the temperature frame of each infrared image based on the temperature gradient value of each temperature frame.
[0039] The infrared image of the temperature frame with the largest average gradient magnitude is selected as the fusion reference image;
[0040] Multiple focal images are identified, which are infrared images of temperature frames with successively decreasing average gradient magnitudes.
[0041] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a method for fusing fully focused infrared images.
[0042] Fourthly, a computer-readable storage medium is provided for storing a computer program thereon, which, when executed by a processor, implements a method for fusing fully focused infrared images.
[0043] The present application will be further described below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0045] Figure 1 This is a flowchart illustrating the all-focused infrared image fusion method in the embodiments of this application;
[0046] Figure 2 This is a block diagram of an electronic device used to implement the fully focused infrared image fusion method in the embodiments of this application; Detailed Implementation
[0047] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.
[0048] like Figure 1 As shown, this embodiment provides a fully focused infrared image fusion method, which is applied in an infrared thermal imager. The method includes steps S100 to S300.
[0049] Step S100: Acquire multiple infrared images. The multiple infrared images have different focal points, and each infrared image has a temperature frame.
[0050] The infrared thermal imager includes an AF motor, which is used to acquire multiple infrared images.
[0051] Step S200: Extract the temperature distribution characteristics from the temperature frame of each infrared image to determine the fusion reference map and multiple focal images;
[0052] Step S300: The full-focus fusion algorithm is used to fuse the fusion reference image and multiple focal images to obtain the full-focus fusion image.
[0053] This fully focused infrared image fusion method can overcome the limitations of traditional infrared thermal imagers in terms of depth of field, and can also improve the detail and clarity of images, making them more visually appealing.
[0054] like Figure 1 As shown, in some embodiments, determining the fusion reference image and multiple focal images includes:
[0055] Obtain the horizontal and vertical temperature change rates of each infrared image's temperature frame;
[0056] The temperature gradient value of each infrared image frame is calculated based on the horizontal and vertical temperature change rates.
[0057] Specifically, the temperature gradient value of each infrared image's temperature frame is calculated using the following formula:
[0058]
[0059] in, The rate of temperature change in the horizontal direction. This represents the rate of temperature change in the vertical direction.
[0060] Calculate the average gradient value of the temperature frame of each infrared image based on the temperature gradient value of each temperature frame.
[0061] Specifically, the average gradient value of the temperature frame for each infrared image is calculated using the following formula:
[0062]
[0063] in, is the temperature gradient value, and N is the number of pixels in the infrared image.
[0064] The infrared image of the temperature frame with the largest average gradient magnitude is selected as the fusion reference image;
[0065] Multiple focal images are identified, which are infrared images of temperature frames with successively decreasing average gradient magnitudes.
[0066] like Figure 1 As shown, in some embodiments, a fully focused fusion map is obtained, including:
[0067] The pre-trained convolutional neural network ResNet is used to extract features from the fused baseline image and multiple focal images. The input is each infrared thermal image grayscale image (1×H×W). The network uses a 2 to 3 layer convolutional + pooling structure to extract local features and outputs multiple sets of semantic feature tensors F1, F2, ... with a shape of C×H×W, where C is the number of channels, H is the image height, and W is the image width.
[0068] For each extracted feature map, ambiguity analysis is performed, and the corresponding ambiguity score map is generated by combining the Sobel gradient method.
[0069] Based on the ambiguity scoring map, a preliminary fusion mask M(x,y) is constructed, and high weights are assigned to clear regions;
[0070] The fusion baseline image and multiple focal images are weighted and superimposed according to the fusion mask. The fusion result is as follows:
[0071]
[0072] Among them, F i (x,y) represents the feature value of the i-th image at position (x,y), M i (x,y) represents the pixel-level fusion weights, F fused This is a fully focused fusion diagram.
[0073] like Figure 1 As shown, in some embodiments, the method further includes adding custom information to the fused full-focus image, such as thermal imager model, serial number, lens parameters, software version, hardware version, extended information, color palette, image mode, isotherm mode, emissivity, reflectance temperature, ambient temperature, relative humidity, target distance, and FPA temperature. Thus, secondary editing is possible after saving the fused full-focus image.
[0074] Secondly, a fully focused infrared image fusion device is provided, comprising:
[0075] The image acquisition module is used to acquire multiple infrared images, each with a different focus and a temperature frame.
[0076] The focus filtering module is used to extract the temperature distribution characteristics of the temperature frame of each infrared image and determine the fusion reference map and multiple focus images;
[0077] The fusion processing module is used to fuse the fusion reference image and multiple focal images using a full-focus fusion algorithm to obtain a full-focus fused image.
[0078] like Figure 1 As shown, in some embodiments, determining the fusion reference image and multiple focal images includes:
[0079] Obtain the horizontal and vertical temperature change rates of each infrared image's temperature frame;
[0080] The temperature gradient value of each infrared image frame is calculated based on the horizontal and vertical temperature change rates.
[0081] Calculate the average gradient value of the temperature frame of each infrared image based on the temperature gradient value of each temperature frame.
[0082] The infrared image of the temperature frame with the largest average gradient magnitude is selected as the fusion reference image;
[0083] Multiple focal images are identified, which are infrared images of temperature frames with successively decreasing average gradient magnitudes.
[0084] To implement the above embodiments, this application provides an electronic device and a computer-readable storage medium.
[0085] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a fully focused infrared image fusion method.
[0086] This embodiment provides a computer-readable storage medium for storing a computer program thereon, which, when executed by a processor, implements a method for fusing fully focused infrared images.
[0087] Figure 2 This is a block diagram of an electronic device used to implement the full-focus infrared image fusion method in an embodiment of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.
[0088] like Figure 2 As shown, the electronic device includes a memory 610, a processor 620, and a computer program 630 stored in the memory and executable on the processor. The various components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processor can process instructions executed within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system).
[0089] The memory 610 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause at least one processor to perform the methods of the above embodiments. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the methods of the above embodiments.
[0090] The memory 610, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the above embodiments. The processor 620 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 610, thereby implementing the methods in the above embodiments.
[0091] The memory 610 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device for implementing the methods in the above embodiments. Furthermore, the memory 610 may include high-speed random access memory and may also include non-transient memory, such as at least one disk storage device, flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 610 may optionally include memory remotely located relative to the processor 620, and these remote memories can be connected via a network to the electronic device for implementing the methods in the above embodiments. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0092] The electronic device used in the methods described in the above embodiments may further include an input device 640 and an output device 650. The processor 620, memory 610, input device 640, and output device 650 may be connected via a bus or other means. Figure 2 Taking the example of a connection between China and Israel via a bus.
[0093] Input device 640 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 650 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.
[0094] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0095] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0096] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0097] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0098] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments. The storage medium mentioned above can be a read-only memory, a magnetic disk, or an optical disk, etc.
[0099] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0100] The above are merely preferred embodiments of this application and do not constitute any limitation on this application. Any person skilled in the art can make many possible variations and modifications to the technical solution of this application, or modify it into equivalent embodiments, without departing from the scope of the technical solution of this application. Therefore, all equivalent changes made based on the shape, structure, and principle of this application without departing from the content of the technical solution of this application should be covered within the protection scope of this application.
Claims
1. A fully focused infrared image fusion method, applied in an infrared thermal imager, characterized in that, The method includes: Multiple infrared images are acquired, each with a different focal point and a temperature frame. Extract the temperature distribution characteristics from the temperature frame of each infrared image to determine a fusion reference map and multiple focal images; wherein, determining the fusion reference map and multiple focal images includes: Obtain the horizontal and vertical temperature change rates of each infrared image's temperature frame; The temperature gradient value of each infrared image temperature frame is calculated based on the horizontal and vertical temperature change rates. Calculate the average gradient value of the temperature frame of each infrared image based on the temperature gradient value of each temperature frame. The infrared image of the temperature frame with the largest average gradient magnitude is selected as the fusion reference image; Multiple focal images are identified, which are infrared images of temperature frames with successively decreasing average gradient amplitudes. A full-focus fusion algorithm is used to fuse the fusion reference image and multiple focal images to obtain a full-focus fusion image; wherein, obtaining the full-focus fusion image includes: The pre-trained convolutional neural network ResNet is used to extract features from the fused baseline image and multiple focal images. The input is each infrared thermal image grayscale image (1×H×W). The network uses a 2 to 3 layer convolution + pooling structure to extract local features and outputs multiple sets of semantic feature tensors F1, F2, ... with a shape of C×H×W, where C is the number of channels, H is the image height, and W is the image width. For each extracted feature map, ambiguity analysis is performed, and the corresponding ambiguity score map is generated by combining the Sobel gradient method. Based on the ambiguity scoring map, a preliminary fusion mask M(x,y) is constructed, and high weights are assigned to clear regions; The fusion baseline image and multiple focal images are weighted and superimposed according to the fusion mask. The fusion result is as follows: ， where F i (x,y) is the feature value of the i-th image at position (x, y), M i (x,y) is the pixel-level fusion weight.
2. The fully focused infrared image fusion method according to claim 1, characterized in that, The infrared thermal imager has an AF motor, and multiple infrared images are acquired by controlling the AF motor.
3. The fully focused infrared image fusion method according to claim 1, characterized in that, The temperature gradient value of each temperature frame in the infrared image is calculated using the following formula: ， in, The rate of temperature change in the horizontal direction. This represents the rate of temperature change in the vertical direction.
4. The fully focused infrared image fusion method according to claim 1, characterized in that, The average gradient value of the temperature frame of each of the infrared images is calculated using the following formula: ， in, is the temperature gradient value, and N is the number of pixels in the infrared image.
5. A fully focused infrared image fusion device, characterized in that, include: An image acquisition module is used to acquire multiple infrared images, wherein the multiple infrared images have different focal points and each infrared image has a temperature frame; The focus filtering module is used to extract the temperature distribution characteristics of the temperature frame of each infrared image to determine the fusion reference map and multiple focus images; wherein, determining the fusion reference map and multiple focus images includes: Obtain the horizontal and vertical temperature change rates of each infrared image's temperature frame; The temperature gradient value of each infrared image temperature frame is calculated based on the horizontal and vertical temperature change rates. Calculate the average gradient value of the temperature frame of each infrared image based on the temperature gradient value of each temperature frame. The infrared image of the temperature frame with the largest average gradient magnitude is selected as the fusion reference image; Multiple focal images are identified, which are infrared images of temperature frames with successively decreasing average gradient amplitudes. The fusion processing module is used to perform fusion processing on the fusion reference image and multiple focal images using a full-focus fusion algorithm to obtain a full-focus fused image, wherein obtaining the full-focus fused image includes: The pre-trained convolutional neural network ResNet is used to extract features from the fused baseline image and multiple focal images. The input is each infrared thermal image grayscale image (1×H×W). The network uses a 2 to 3 layer convolution + pooling structure to extract local features and outputs multiple sets of semantic feature tensors F1, F2, ... with a shape of C×H×W, where C is the number of channels, H is the image height, and W is the image width. For each extracted feature map, ambiguity analysis is performed, and the corresponding ambiguity score map is generated by combining the Sobel gradient method. Based on the ambiguity scoring map, a preliminary fusion mask M(x,y) is constructed, and high weights are assigned to clear regions; The fusion baseline image and multiple focal images are weighted and superimposed according to the fusion mask. The fusion result is as follows: ， Among them, F i (x,y) represents the feature value of the i-th image at position (x,y), M i (x,y) represents the pixel-level fusion weights.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 4.
7. A computer-readable storage medium for storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.