Full-focus infrared image fusion method and device, electronic equipment and medium

Abstract

The invention belongs to the technical field of infrared image processing, and particularly relates to a full-focus infrared image fusion method and device, electronic equipment and a medium, the method is applied to a thermal infrared imager, the method comprises the steps that multiple infrared images are acquired, the focuses of the multiple infrared images are different, and each infrared image has a temperature frame; extracting temperature distribution characteristics of the temperature frame of each infrared image, and determining a fusion reference image and a plurality of focus images; and carrying out fusion processing on the fusion reference image and the plurality of focus images by adopting a full-focus fusion algorithm to obtain a full-focus fusion image. The full-focus infrared image fusion method can break through the limitation of the depth of field of a traditional thermal infrared imager, and can also improve the details and definition of the image, so that the image is more beautiful visually.

Description

Technical Field

[0001] The present application relates to the field of infrared image processing technology, and in particular to a full-focus infrared image fusion method, device, electronic equipment and medium. Background Art

[0002] Infrared thermal imagers are widely used in many fields such as industry, medical treatment, security, and environmental protection. However, when users use infrared thermal imagers to take pictures, it is difficult to ensure that every detail of the image is clear due to the depth of field problem of the infrared thermal imager.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in this field.

[0004] Application Contents

[0005] In view of at least one of the above technical problems, the present application provides a full-focus infrared image fusion method, device, electronic device and medium.

[0006] In a first aspect, a method for fusion of all-focus infrared images is provided, which is applied to an infrared thermal imager. The method comprises:

[0007] Acquire multiple infrared images, where the multiple infrared images have different focuses and each infrared image has a temperature frame;

[0008] Extract the temperature distribution characteristics of each infrared image's temperature frame and determine the fusion reference image and multiple focus images;

[0009] The full-focus fusion algorithm is used to fuse the fusion reference image and multiple focus images to obtain the full-focus fusion image.

[0010] This all-focus infrared image fusion method can break through the limitations of the depth of field of traditional infrared thermal imagers, and can also improve the details and clarity of the image, making the image more visually beautiful.

[0011] In some possible implementations, the infrared thermal imager has an AF motor, and multiple infrared images are acquired by controlling the AF motor.

[0012] In some possible implementations, determining to fuse the reference image and the multiple focus images includes:

[0013] Obtain the horizontal temperature change rate and vertical temperature change rate of the temperature frame of each infrared image;

[0014] Calculate the temperature gradient value of the temperature frame of each infrared image based on the horizontal temperature change rate and the vertical temperature change rate;

[0015] Calculating an average gradient value of the temperature frame of each infrared image according to the temperature gradient value of the temperature frame of each infrared image;

[0016] Determine the infrared image of the temperature frame with the largest average gradient amplitude as the fusion reference image;

[0017] A plurality of focus images are determined, where the plurality of focus images are infrared images of temperature frames with average gradient amplitudes decreasing in sequence.

[0018] In some possible implementations, the temperature gradient value of the temperature frame of each infrared image is calculated using the following formula:

[0019]

[0020] in, is the temperature change rate in the horizontal direction, is the vertical temperature change rate.

[0021] In some possible implementations, the average gradient value of the temperature frame of each infrared image is calculated using the following formula:

[0022]

[0023] in, is the temperature gradient value, and N is the number of pixels in the infrared image.

[0024] In some possible implementations, obtaining a fully focused fusion image includes:

[0025] A pre-trained convolutional neural network (ResNet) is used to extract features from the fused reference image and multiple focus images. The input is each infrared thermal image grayscale image (1×H×W). The network uses a 2- to 3-layer convolution + pooling structure to extract local features and outputs multiple sets of semantic feature tensors F1, F2, ..., with a shape of C×H×W, where C is the number of channels, H is the image height, and W is the image width.

[0026] Perform fuzziness analysis on each set of extracted feature maps and generate the corresponding fuzziness score map by combining the Sobel gradient method;

[0027] Based on the fuzziness score map, a preliminary fusion mask M(x,y) is constructed, giving high weights to clear areas;

[0028] The fused reference image and multiple focus images are weightedly superimposed according to the fusion mask, and the fusion result is:

[0029]

[0030] Among them, F i (x,y) is the eigenvalue of the i-th image at position (x,y), Mi (x,y) is the pixel-level fusion weight.

[0031] In a second aspect, a full-focus infrared image fusion device is provided, comprising:

[0032] An image acquisition module is used to acquire multiple infrared images, where the multiple infrared images have different focuses and each infrared image has a temperature frame;

[0033] The focus screening module is used to extract the temperature distribution characteristics of the temperature frame of each infrared image and determine the fusion reference image and multiple focus images;

[0034] The fusion processing module is used to fuse the fusion reference image and multiple focus images using the full-focus fusion algorithm to obtain the full-focus fusion image.

[0035] In some possible implementations, determining to fuse the reference image and the multiple focus images includes:

[0036] Obtain the horizontal temperature change rate and vertical temperature change rate of the temperature frame of each infrared image;

[0037] Calculate the temperature gradient value of the temperature frame of each infrared image based on the horizontal temperature change rate and the vertical temperature change rate;

[0038] Calculating an average gradient value of the temperature frame of each infrared image according to the temperature gradient value of the temperature frame of each infrared image;

[0039] Determine the infrared image of the temperature frame with the largest average gradient amplitude as the fusion reference image;

[0040] A plurality of focus images are determined, where the plurality of focus images are infrared images of temperature frames with average gradient amplitudes decreasing in sequence.

[0041] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, a full-focus infrared image fusion method is implemented.

[0042] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, a full-focus infrared image fusion method is implemented.

[0043] The present application is further described below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 Schematic diagram of the process of the all-focus infrared image fusion method in the embodiment of the present application;

[0046] Figure 2 This is a block diagram of an electronic device for implementing the full-focus infrared image fusion method in an embodiment of the present application; DETAILED DESCRIPTION

[0047] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0048] like Figure 1 As shown, this embodiment provides a full-focus infrared image fusion method, which is applied to an infrared thermal imager. The method includes: steps S100 to S300.

[0049] Step S100 : acquiring a plurality of infrared images, wherein the focus of the plurality of infrared images is different and each infrared image has a temperature frame.

[0050] The infrared thermal imager has an AF motor, which is used to control the AF motor to obtain multiple infrared images.

[0051] Step S200 , extracting temperature distribution characteristics from the temperature frame of each infrared image, and determining a fusion reference image and multiple focus images;

[0052] Step S300: Using an all-focus fusion algorithm to fuse the fusion reference image and multiple focus images to obtain an all-focus fusion image.

[0053] This all-focus infrared image fusion method can break through the limitations of the depth of field of traditional infrared thermal imagers, and can also improve the details and clarity of the image, making the image more visually beautiful.

[0054] like Figure 1 As shown, in some embodiments, determining to fuse the reference image and the multiple focus images includes:

[0055] Obtain the horizontal temperature change rate and vertical temperature change rate of the temperature frame of each infrared image;

[0056] Calculate the temperature gradient value of the temperature frame of each infrared image based on the horizontal temperature change rate and the vertical temperature change rate;

[0057] Specifically, the temperature gradient value of the temperature frame of each infrared image is calculated using the following formula:

[0058]

[0059] in, is the temperature change rate in the horizontal direction, is the vertical temperature change rate.

[0060] Calculating an average gradient value of the temperature frame of each infrared image according to the temperature gradient value of the temperature frame of each infrared image;

[0061] Specifically, the average gradient value of the temperature frame of each infrared image is calculated using the following formula:

[0062]

[0063] in, is the temperature gradient value, and N is the number of pixels in the infrared image.

[0064] Determine the infrared image of the temperature frame with the largest average gradient amplitude as the fusion reference image;

[0065] A plurality of focus images are determined, where the plurality of focus images are infrared images of temperature frames with average gradient amplitudes decreasing in sequence.

[0066] like Figure 1 As shown, in some embodiments, obtaining a fully focused fusion image includes:

[0067] A pre-trained convolutional neural network (ResNet) is used to extract features from the fused reference image and multiple focus images. The input is each infrared thermal image grayscale image (1×H×W). The network uses a 2- to 3-layer convolution + pooling structure to extract local features and outputs multiple sets of semantic feature tensors F1, F2, ..., with a shape of C×H×W, where C is the number of channels, H is the image height, and W is the image width.

[0068] Perform fuzziness analysis on each set of extracted feature maps and generate the corresponding fuzziness score map by combining the Sobel gradient method;

[0069] Based on the fuzziness score map, a preliminary fusion mask M(x,y) is constructed, giving high weights to clear areas;

[0070] The fused reference image and multiple focus images are weightedly superimposed according to the fusion mask, and the fusion result is:

[0071]

[0072] Among them, F i (x,y) is the eigenvalue of the i-th image at position (x,y), M i (x,y) is the pixel-level fusion weight, F fused This is a fully focused fusion image.

[0073] like Figure 1 As shown, in some embodiments, the method further includes: adding custom information to the fused all-focus fusion image, including parameters such as thermal imager model, serial number, lens parameters, software version, hardware version, extended information, color palette, image mode, isotherm mode, emissivity, reflected temperature, ambient temperature, relative humidity, target distance, FPA temperature, etc. In this way, after saving the all-focus fusion image, it can be edited again.

[0074] In a second aspect, a full-focus infrared image fusion device is provided, comprising:

[0075] An image acquisition module is used to acquire multiple infrared images, where the multiple infrared images have different focuses and each infrared image has a temperature frame;

[0076] The focus screening module is used to extract the temperature distribution characteristics of the temperature frame of each infrared image and determine the fusion reference image and multiple focus images;

[0077] The fusion processing module is used to fuse the fusion reference image and multiple focus images using the full-focus fusion algorithm to obtain the full-focus fusion image.

[0078] like Figure 1 As shown, in some embodiments, determining to fuse the reference image and the multiple focus images includes:

[0079] Obtain the horizontal temperature change rate and vertical temperature change rate of the temperature frame of each infrared image;

[0080] Calculate the temperature gradient value of the temperature frame of each infrared image based on the horizontal temperature change rate and the vertical temperature change rate;

[0081] Calculating an average gradient value of the temperature frame of each infrared image according to the temperature gradient value of the temperature frame of each infrared image;

[0082] Determine the infrared image of the temperature frame with the largest average gradient amplitude as the fusion reference image;

[0083] A plurality of focus images are determined, where the plurality of focus images are infrared images of temperature frames with average gradient amplitudes decreasing in sequence.

[0084] In order to implement the above embodiments, the present application provides an electronic device and a computer-readable storage medium.

[0085] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, a full-focus infrared image fusion method is implemented.

[0086] This embodiment provides a computer-readable storage medium for storing a computer program thereon, and when the computer program is executed by a processor, a full-focus infrared image fusion method is implemented.

[0087] Figure 2 This is a block diagram of an electronic device for implementing the all-focus infrared image fusion method in an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided for example only and are not intended to limit the implementation of the present application as described and / or claimed herein.

[0088] like Figure 2 As shown, the electronic device includes: a memory 610, a processor 620 and a computer program 630 stored in the memory and operable on the processor. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0089] Memory 610 is a non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the method of the above embodiment. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the method of the above embodiment.

[0090] Memory 610, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the above-described embodiments. Processor 620 executes the non-transitory software programs, instructions, and modules stored in memory 610 to execute various server functional applications and data processing, thereby implementing the methods in the above-described embodiments.

[0091] The memory 610 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the electronic device used to implement the method in the above embodiment, etc. In addition, the memory 610 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 610 may optionally include a memory remotely located relative to the processor 620, and these remote memories may be connected to the electronic device used to implement the method in the above embodiment via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0092] The electronic device used in the method of the above embodiment may further include: an input device 640 and an output device 650. The processor 620, the memory 610, the input device 640 and the output device 650 may be connected via a bus or other means. Figure 2 The bus connection is taken as an example.

[0093] The input device 640 can receive input digital or character information and generate key signal input related to user settings and function control of the electronic device, such as input devices such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, a pointer, one or more mouse buttons, a trackball, and a joystick. The output device 650 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0094] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0095] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0096] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0097] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0098] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be performed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps in the method embodiments. The aforementioned storage medium can be a read-only memory, a magnetic disk, or an optical disk.

[0099] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0100] The above are merely preferred embodiments of the present application and do not constitute any form of limitation to the present application. Any person skilled in the art can, without departing from the scope of the technical solution of the present application, use the methods and technical contents disclosed above to make many possible changes and modifications to the technical solution of the present application, or modify it into an equivalent embodiment with equivalent changes. Therefore, all equivalent changes made based on the shape, structure and principle of the present application without departing from the content of the technical solution of the present application should be included in the scope of protection of the present application.

Claims

1. A full-focus infrared image fusion method, applied to an infrared thermal imager, characterized in that: The method comprises: Acquiring a plurality of infrared images, wherein the plurality of infrared images have different focuses and each of the infrared images has a temperature frame; extracting temperature distribution characteristics from the temperature frame of each infrared image, and determining a fusion reference image and multiple focus images; The fusion reference image and the multiple focus images are fused using a full focus fusion algorithm to obtain a full focus fusion image.

2. The all-focus infrared image fusion method according to claim 1, characterized in that: The infrared thermal imager has an AF motor, and multiple infrared images are acquired by controlling the AF motor.

3. The all-focus infrared image fusion method according to claim 1, characterized in that: The determining and fusing the reference image and the multiple focus images includes: Acquire the horizontal temperature change rate and the vertical temperature change rate of the temperature frame of each infrared image; Calculating a temperature gradient value of a temperature frame of each infrared image according to the horizontal temperature change rate and the vertical temperature change rate; Calculating an average gradient value of the temperature frame of each infrared image according to the temperature gradient value of the temperature frame of each infrared image; Determine the infrared image of the temperature frame with the largest average gradient amplitude as the fusion reference image; A plurality of focus images are determined, where the plurality of focus images are infrared images of temperature frames with decreasing average gradient amplitudes.

4. The all-focus infrared image fusion method according to claim 3, characterized in that: The temperature gradient value of each temperature frame of the infrared image is calculated using the following formula: in, is the temperature change rate in the horizontal direction, is the vertical temperature change rate.

5. The all-focus infrared image fusion method according to claim 3, characterized in that: The average gradient value of the temperature frame of each infrared image is calculated using the following formula: in, is the temperature gradient value, and N is the number of pixels in the infrared image.

6. The all-focus infrared image fusion method according to claim 1, characterized in that: The obtaining of the fully focused fusion image comprises: A pre-trained convolutional neural network (ResNet) is used to extract features from the fused reference image and multiple focus images. The input is each infrared thermal image grayscale image (1×H×W). The network uses a 2- to 3-layer convolution + pooling structure to extract local features and outputs multiple sets of semantic feature tensors F1, F2, ..., with a shape of C×H×W, where C is the number of channels, H is the image height, and W is the image width. Perform fuzziness analysis on each set of extracted feature maps and generate the corresponding fuzziness score map by combining the Sobel gradient method; Based on the fuzziness score map, a preliminary fusion mask M(x,y) is constructed, giving high weights to clear areas; The fused reference image and multiple focus images are weightedly superimposed according to the fusion mask, and the fusion result is: Among them, F i (x,y) is the eigenvalue of the i-th image at position (x,y), M i (x,y) is the pixel-level fusion weight.

7. A fully focused infrared image fusion device, characterized in that: include: An image acquisition module, configured to acquire a plurality of infrared images, wherein the plurality of infrared images have different focuses and each of the infrared images has a temperature frame; A focus screening module, configured to extract temperature distribution characteristics from the temperature frame of each infrared image, and determine a fusion reference image and multiple focus images; The fusion processing module is used to use an all-focus fusion algorithm to fuse the fusion reference image and multiple focus images to obtain an all-focus fusion image.

8. The all-focus infrared image fusion device according to claim 1, characterized in that: The determining and fusing the reference image and the multiple focus images includes: Acquire the horizontal temperature change rate and the vertical temperature change rate of the temperature frame of each infrared image; Calculating a temperature gradient value of a temperature frame of each infrared image according to the horizontal temperature change rate and the vertical temperature change rate; Calculating an average gradient value of the temperature frame of each infrared image according to the temperature gradient value of the temperature frame of each infrared image; Determine the infrared image of the temperature frame with the largest average gradient amplitude as the fusion reference image; A plurality of focus images are determined, where the plurality of focus images are infrared images of temperature frames with decreasing average gradient amplitudes.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images