Image fusion method, device, and storage medium

Abstract

This application provides an image fusion method, apparatus, and storage medium. The image fusion method includes dividing a registered infrared image and a visible light image into multiple image blocks, then transforming each image block to the frequency domain using Fourier transform, fusing the image blocks in the frequency domain, and finally merging the fused image blocks into a fused image. Therefore, the fused image obtained using this image fusion method effectively distinguishes and extracts the effective information from the infrared and visible light images, improving the utilization rate of image information.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image fusion method, apparatus, and storage medium. Background Technology

[0002] Infrared imaging technology uses infrared radiation to perceive the surrounding environment, and its images are characterized by good environmental adaptability, good concealment, and high recognition of camouflaged targets. Thanks to its imaging principle, infrared images possess temperature characterization properties; with certain algorithmic processing, the temperature distribution information of a target can be perceived. However, limited by the manufacturing difficulty and material purity of infrared detectors, infrared images generally suffer from low resolution, high noise, low image contrast, and narrow grayscale range. This results in blurred contrast between the background and the monitored target, making it difficult to discern details of the monitored target and unclear image feature information, thus posing challenges to the identification and analysis of targets and scenes in the images. Compared to infrared images, visible light images have higher contrast and resolution, and offer advantages such as rich spectral information, large dynamic range, abundant detail, and good visual appeal. However, visible light images have poor anti-interference capabilities; in low light, foggy conditions, or when targets are camouflaged, the effect of visible light images becomes significantly unsatisfactory, making it difficult to obtain information about targets and scenes, and even making it impossible to analyze and identify targets and scenes. Therefore, infrared and visible light images are two types of images with complementary advantages and disadvantages.

[0003] Simultaneously observing a scene or object with both an infrared camera and a visible light camera allows for a more comprehensive understanding of its characteristics under a wider range of environmental conditions. However, in practical applications, using two separate displays to show infrared and visible light images would waste display resources, while using a single display to divide the infrared and visible light images into two parts would reduce the image display quality. Therefore, image fusion technology, which combines infrared and visible light images into a single image, has emerged.

[0004] Patent application CN109478315A discloses an image fusion optimization system and method, which extracts and utilizes only the contour / edge information in the visible light image during the fusion of infrared and visible light images, while ignoring other large-scale information provided by the visible light image. Furthermore, patent application CN105069768A discloses a visible light and infrared image fusion processing system and method, whose image fusion scheme involves a series of operations such as low-pass filtering, detail enhancement, low-frequency fusion, and high-low frequency fusion. The overall process is overly cumbersome and difficult to output video streams in real time on devices with limited computing power. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides an image fusion method, apparatus, and storage medium that can acquire more detailed information and streamline the process.

[0006] An image fusion method, comprising:

[0007] The registered infrared and visible light images are divided into multiple image blocks to obtain the corresponding infrared and visible light image blocks;

[0008] Perform Fourier transform on each of the infrared image blocks and the visible light image blocks to obtain the corresponding frequency domain infrared image blocks and frequency domain visible light image blocks;

[0009] Each frequency domain infrared image block is fused with its corresponding frequency domain visible light image block according to a preset fusion rule to obtain a corresponding frequency domain fused image block.

[0010] Perform an inverse Fourier transform on each of the frequency domain fused image blocks to obtain the corresponding spatial domain fused image blocks;

[0011] The individual spatial fusion image blocks are merged to obtain a fused image.

[0012] An image fusion device, including a memory and a processor;

[0013] When the processor executes the computer program instructions stored in the memory, it performs the steps of the image fusion method.

[0014] A computer-readable storage medium storing computer program instructions;

[0015] When the computer program instructions are executed by the processor, the steps of the image fusion method are implemented.

[0016] As can be seen from the above, in the image fusion method, apparatus, and storage medium provided in this application, the registered infrared and visible light images are divided into multiple image blocks. Then, each image block is transformed to the frequency domain using Fourier transform, and the image blocks are fused in the frequency domain. Finally, the fused image blocks are merged into a fused image. Therefore, the fused image obtained using the image fusion method described above effectively distinguishes and extracts the effective information from the infrared and visible light images, improving the utilization rate of image information. Attached Figure Description

[0017] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0018] Figure 1This is a schematic flowchart of an image fusion method provided according to some embodiments of this application;

[0019] Figure 2 This is a schematic flowchart of an image fusion method provided according to other embodiments of this application;

[0020] Figure 3 This is a schematic diagram illustrating the processing steps of an image fusion method provided according to some embodiments of this application;

[0021] Figure 4 This is a schematic diagram of the structure of an image fusion apparatus provided according to some embodiments of this application;

[0022] Figure 5 This is a schematic diagram of the structure of an image fusion device provided according to some embodiments of this application. Detailed Implementation

[0023] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to limit the ways in which this application may be implemented. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0025] In the description of this application, it should be understood that the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "row," "top," "bottom," "inner," and "outer," etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings and are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0026] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0027] Figure 1This is a flowchart illustrating an image fusion method provided in some embodiments of this application. The image fusion method uses... Figure 4 The image fusion device shown or Figure 5 The image fusion device shown implements this. Specifically, the image fusion method includes steps S2, S4, S6, S8, and S10.

[0028] S2: Divide the registered infrared image and visible light image into multiple image blocks to obtain the corresponding infrared image blocks and visible light image blocks.

[0029] S2 can be derived from Figure 4 The image segmentation module 11 in the illustrated image device is implemented, or is implemented by... Figure 5 The image fusion device shown has a memory 21 that stores the image segmentation program, which is then processed by... Figure 5 The processor 22 in the fused image device shown is implemented when executing the image segmentation program stored in the memory 21.

[0030] The registered infrared and visible light images mentioned in S2 refer to the original infrared and visible light images after being aligned and mapped to the same coordinate system. The registered infrared and visible light images are divided into blocks according to their location regions using an image segmentation algorithm. This divides the registered infrared image into multiple infrared image blocks and the registered visible light image into a corresponding number of visible light image blocks. That is, for each infrared image block, there is a corresponding visible light image block. Here, "corresponding in location" means that the infrared image block and the corresponding visible light image block are positioned relative to each other in the same coordinate system. By segmenting the registered infrared and visible light images into smaller image blocks, the large original infrared and visible light images are reduced, thus decreasing the amount of data and computational burden during each image fusion operation.

[0031] S4: Perform Fourier transform on each of the infrared image blocks and the visible light image blocks to obtain the corresponding frequency domain infrared image blocks and frequency domain visible light image blocks.

[0032] S4 can be derived from Figure 4 The Fourier transform module 12 in the image device shown is implemented, or is implemented by... Figure 5 The image fusion device shown has a memory 21 that stores the Fourier transform program, which is then processed by... Figure 5 The processor 22 in the fused image device shown is implemented when executing the Fourier transform program stored in the memory 21.

[0033] Performing a two-dimensional Fourier transform on each infrared and visible light image block transforms them from the spatial domain to the frequency domain. Fusion of the images in the frequency domain allows for better differentiation and extraction of key information from both the infrared and visible light images. This is because, in the frequency domain, the image is decomposed into low-frequency components representing overall information and high-frequency components representing local details. During the fusion of infrared and visible light images, it is necessary to preserve as much overall information as possible in the infrared image while neglecting its details, and to focus as much as possible on the details of the visible light image while neglecting its overall information. Therefore, decomposing each infrared and visible light image block into the frequency domain is beneficial for differentiating and extracting effective information from the infrared and visible light images.

[0034] S6: Fuse each frequency domain infrared image block with its corresponding frequency domain visible light image block according to a preset fusion rule to obtain the corresponding frequency domain fused image block.

[0035] S6 can be generated by Figure 4 The image fusion module 13 in the illustrated image device is implemented, or is implemented by... Figure 5 The image fusion device shown has a memory 21 that stores the image fusion program, which is then processed by... Figure 5 The processor 22 in the fused image device shown is implemented when executing the image fusion program stored in the memory 21.

[0036] The correspondence between the frequency domain infrared image block and the frequency domain visible light image block means that the position of the infrared image block corresponding to the frequency domain infrared image block in the same coordinate system corresponds to the position of the visible light image block corresponding to the frequency domain visible light image block in the same coordinate system.

[0037] If, in S2, the registered infrared image is divided into K infrared image blocks, then the corresponding registered visible light image is also divided into K visible light image blocks. There is a one-to-one correspondence between the K infrared image blocks and the K visible light image blocks. If the nth infrared image block and the mth visible light image block in the same coordinate system have the same corresponding position, then the visible light image block corresponding to the nth infrared image block is the mth visible light image block. Here, n and m can be the same or different. In S4, the nth infrared image block undergoes a Fourier transform to obtain the corresponding nth frequency domain infrared image block, and the mth visible light image block undergoes a Fourier transform to obtain the corresponding mth frequency domain visible light image block. Then, in S6, the nth frequency domain infrared image block and the mth frequency domain visible light image block are fused according to a preset fusion rule to obtain the corresponding frequency domain fused image block.

[0038] S8: Perform inverse Fourier transform on each of the frequency domain fused image blocks to obtain the corresponding spatial domain fused image blocks.

[0039] S8 can be Figure 4 The inverse Fourier transform module 14 in the image device shown is implemented, or is implemented by... Figure 5 The memory 21 in the image fusion device shown stores the inverse Fourier transform program, which is then processed by... Figure 5 The processor 22 in the fused image device shown is implemented when executing the inverse Fourier transform program stored in the memory 21.

[0040] After the fusion of the corresponding frequency domain image blocks is completed, the fused frequency domain image blocks are transformed back into the spatial domain using a two-dimensional inverse Fourier transform, so that the fused image blocks can be merged in the spatial domain later.

[0041] S10: Merge the various spatial fusion image blocks to obtain a fused image.

[0042] S10 can be... Figure 4 The image merging module 15 in the illustrated image device is implemented, or is implemented by... Figure 5 The image fusion device shown has a memory 21 that stores the image merging program, which is then processed by... Figure 5 The processor 22 in the fused image device shown is implemented when executing the image merging program stored in the memory 21.

[0043] During the merging of various spatial fusion image blocks, it is necessary to restore a complete fusion image according to the original order based on the corresponding positions of each spatial fusion image block in the same coordinate system.

[0044] As can be seen above, the registered infrared and visible light images are divided into multiple image blocks. Then, each image block is transformed into the frequency domain using Fourier transform. These image blocks are then fused in the frequency domain, and finally, they are merged into a fused image. Therefore, the fused image obtained using the image fusion method provided in this application effectively distinguishes and extracts the useful information from the infrared and visible light images, improving the utilization rate of image information.

[0045] In some embodiments, S2 specifically involves uniformly dividing the registered infrared image and visible light image into multiple image blocks along the row and column directions, respectively, to obtain corresponding infrared image blocks and visible light image blocks. For example, the registered infrared image and visible light image are each divided into N×M corresponding image blocks, that is, the registered infrared image and visible light image are each divided into N rows and M columns, with the M image blocks in each row having equal areas and the N image blocks in each column having equal areas.

[0046] Specifically, in some embodiments, a sliding window of a preset size can be used to sequentially traverse the configured infrared and visible light images at preset step sizes. Each step of the sliding window obtains a corresponding infrared image block and a visible light image block. In other embodiments, the required number or size of image blocks can be defined, and then the images are divided starting from their respective pixel origins according to the sizes of the registered infrared and visible light images to obtain the corresponding infrared and visible light image blocks. The specific method of dividing the registered infrared and visible light images into image blocks is not particularly limited in this application.

[0047] Furthermore, in some embodiments, the registered infrared and visible light images are uniformly divided into multiple image blocks along the row and column directions, respectively, to obtain corresponding infrared and visible light image blocks. Adjacent infrared and visible light image blocks have a predetermined overlap region. Setting an appropriately sized overlap region between adjacent image blocks can reduce the "pseudo-structure" that may be introduced in the frequency domain fusion step (S6), thus avoiding image distortion during fusion.

[0048] In other embodiments, the registered infrared and visible light images are uniformly divided into multiple image blocks along the row and column directions, respectively, to obtain corresponding infrared and visible light image blocks. There is no overlap between any adjacent infrared image blocks or adjacent visible light image blocks. Alternatively, for areas prone to "pseudo-structure" (distortion) during fusion, adjacent infrared and visible light image blocks are configured to have a preset overlap area, while in the remaining areas, there is no overlap between any adjacent infrared and visible light image blocks.

[0049] There are many possible preset fusion rules in S6. For example, the result of a pixel-by-pixel weighted average of the frequency domain infrared image block and the corresponding frequency domain visible light image block according to a preset weight can be used as the corresponding frequency domain fused image block. The preset weight can be set based on the corresponding pixel positions of the frequency domain infrared and visible light image blocks. Alternatively, during the fusion of the corresponding frequency domain image blocks, pixels with larger absolute intensity values ​​in the infrared and visible light image blocks can be selected as the corresponding pixels in the frequency domain fused image block. Or, pixels from the frequency domain infrared image block can be selected in the low-frequency band, while pixels from the frequency domain visible light image block can be selected in the high-frequency band to determine the pixels in the corresponding frequency domain fused image. In this application, the method of fusion between corresponding frequency domain image blocks is not particularly limited.

[0050] Therefore, in some embodiments, S6 specifically includes: weighting each frequency-domain infrared image block and its corresponding frequency-domain visible light image block pixel by pixel according to a preset weight to obtain a corresponding frequency-domain fused image block. The preset weight is set based on the corresponding pixel position of the corresponding frequency-domain image block; for example, the farther the corresponding pixel is from the origin, the smaller its preset weight. Weighting pixel by pixel according to the preset weight includes weighting the pixels in the frequency-domain infrared image block with a first weight to determine a first weighted value, and weighting the pixels at the corresponding position in the frequency-domain visible light image block with a second weight to determine a second weighted value. Then, the average of the first and second weighted values ​​is determined, and this average is used as the pixel at the corresponding position in the frequency-domain fused image block. The sum of the first and second weights is 1.

[0051] In some embodiments, S6 may specifically include: comparing the pixel values ​​of each frequency-domain infrared image block with the pixel values ​​at the same position in its corresponding frequency-domain visible light image, determining the pixel at the corresponding position based on the pixel with the larger pixel value, and obtaining the corresponding frequency-domain fused image block. For example, if pixel A in the frequency-domain infrared image block and pixel B in the corresponding visible light image block are pixels at the same position, then the pixel values ​​of the two pixels are compared. If the pixel value of pixel A is larger, then the pixel of pixel A is taken as the pixel at the corresponding position in the frequency-domain fused image block; otherwise, pixel B is taken as the pixel at the corresponding position in the frequency-domain fused image block. Here, "corresponding position" refers to a position with the same pixel coordinates.

[0052] Furthermore, since the low-frequency portion in the frequency domain primarily represents overall information, while the high-frequency portion primarily represents local detail information, in order to preserve as much overall information as possible in the infrared image while ignoring the detailed information of the infrared image, and to focus as much on the detailed information of the visible light image as possible while ignoring the overall information of the visible light image, in some embodiments, S6 may further include: determining the corresponding pixel position based on the pixels in the frequency domain infrared image block in the low-frequency band, and determining the corresponding pixel position based on the pixels in the frequency domain visible light image block in the high-frequency band, to obtain the corresponding frequency domain fusion image block. In other embodiments, S6 may also employ a fusion rule that weights each frequency domain infrared image block and its corresponding frequency domain visible light image block pixel by pixel according to a preset weight to obtain the corresponding frequency domain fusion image block. However, the preset weight here is related to the frequency band; in the low-frequency band, the first weight is greater than the second weight, while in the high-frequency band, the first weight is less than the second weight.

[0053] During the image block partitioning process, if adjacent image blocks are set to have a preset overlapping region, then corresponding adjacent spatial fusion image blocks also have corresponding overlapping regions. That is, each spatial fusion image block in an adjacent spatial fusion image block consists of a non-overlapping region and an overlapping region. In some embodiments, S10 specifically includes: during the sequential merging of various spatial fusion image blocks, determining the pixels at the corresponding positions (positions corresponding to the non-overlapping regions) in the fused image based on the pixels in the non-overlapping regions of the spatial fusion image blocks, and determining the pixels at the corresponding positions (positions corresponding to the overlapping regions) in the fused image based on the average pixels of adjacent spatial fusion image blocks with overlapping regions in the corresponding overlapping regions. If spatial fusion image block a and spatial fusion image block b are adjacent and their regions overlap by region d, then the pixels at the positions corresponding to the overlapping region d in the fused image are determined by the average pixels of spatial fusion image blocks a and b in the overlapping region d. In other embodiments, the pixel at the position corresponding to the overlapping region d in the fused image can also be determined by comparing the distances of the center pixels of two adjacent spatial fused image blocks a and b from the overlapping region d, and determining the pixel at the position corresponding to the overlapping region d in the fused image based on the overlapping region pixels of the spatial fused image block with the closer distance. For example, if the center position of spatial fused image block a is relatively close to the position of overlapping region d, then the pixels of the overlapping region of spatial fused image block a are taken as the pixels at the position corresponding to the overlapping region d in the fused image.

[0054] like Figure 2 As shown, it is a flowchart illustrating an image fusion method provided according to other embodiments of this application, compared to Figure 1 The image fusion method shown in this embodiment further includes acquiring a registered infrared image and a visible light image. Figure 4 The corresponding fusion device also includes a registration acquisition module, used to acquire the registered infrared and visible light images. This step can also be performed by... Figure 4 The image fusion device in the image fusion device is used to achieve this. The steps of acquiring the registered infrared image and visible light image specifically include S11, S12, S13 and S14.

[0055] S11: Acquire infrared and visible light images of the same target in the same scene.

[0056] In some embodiments of this application, optical devices (such as infrared thermal imagers) equipped with both infrared and visible light cameras can be used to simultaneously capture images of the same target in a scene to obtain corresponding infrared and visible light images. Alternatively, separate infrared and visible light cameras can be used to capture images of the same target in the same scene simultaneously or at different times to obtain corresponding infrared and visible light images. When using different devices to acquire infrared and visible light images, the infrared and visible light cameras can be placed in the same position with their lens optical axes aligned and parallel to obtain infrared and visible light images from the same angle. The infrared and visible light cameras can also be placed in different positions to acquire infrared and visible light images from different angles. The distribution rates of the infrared and visible light images can be the same or different. The resolution of the acquired images can be set when the corresponding capturing device acquires infrared and visible light images, or the resolution can be adjusted after the corresponding images are acquired. Furthermore, after acquiring the corresponding image from the image capturing device (infrared camera, visible light camera), appropriate image processing can be performed on the corresponding image, such as image cropping or image stretching, so that the infrared image and the visible light image shown are the same size.

[0057] S12: Convert the visible light image into a grayscale image.

[0058] In the subsequent image fusion process, detailed information, such as the outline of the target object, is mainly extracted from the visible light image. In order to simplify the subsequent coordinate mapping relationship and the computational complexity of the Fourier transform, the visible light image is grayscale processed in this embodiment to obtain its corresponding grayscale image.

[0059] S13: Determine the coordinate mapping relationship between the grayscale images of the infrared image and the visible light image.

[0060] S14: Align and map the infrared image and the visible light image to the same coordinate system according to the coordinate mapping relationship to obtain the registered infrared image and the visible light image.

[0061] Because infrared and visible light cameras have different focal lengths, fields of view, and resolutions, pixel-by-pixel matching and registration are required during the fusion of infrared and visible light images. This ensures that the same pixel corresponds to the same object, preventing distortion and artifacts in the fused image. The coordinate systems of infrared and visible light images may differ; pixels corresponding to the same target point in the infrared and visible light images display different coordinates in different spatial coordinate systems. Therefore, a coordinate mapping relationship can be established between the two. This mapping relationship can be expressed as follows: the pixel in row i1, column j1 of the infrared image corresponds to the pixel in row i2, column j2 of the visible light image, or vice versa. The coordinate mapping relationship can also be represented by a mapping table; however, this application does not specifically limit the specific form of the coordinate mapping relationship.

[0062] Specifically, in some embodiments, the coordinate mapping relationship between the grayscale images of the infrared image and the visible light image can be determined based on the parameter information (internal parameters) and relative geometric position information of the infrared camera that acquires the infrared image and the visible light camera that acquires the visible light image.

[0063] Furthermore, in some embodiments, aligning the infrared image and the visible light image to the same coordinate system according to the coordinate mapping relationship to obtain the registered infrared image and the visible light image includes: aligning the infrared image and the visible light image to the same coordinate system according to the mapping relationship, and assigning values ​​to pixels corresponding to non-integer pixel coordinates in the same coordinate system based on a grayscale interpolation method. This is because the mapping relationship determined above contains cases where pixel coordinates are not integers; that is, when a pixel in the aligned and mapped image is projected onto the original infrared or visible light image, its corresponding coordinate is not an integer. In some embodiments, the grayscale interpolation method is a nearest neighbor interpolation method. Assigning values ​​to pixels corresponding to non-integer pixel coordinates in the same coordinate system based on the nearest neighbor interpolation method specifically means taking the pixel with the nearest integer coordinate to the non-integer coordinate as the corresponding pixel in the aligned and mapped image. In other embodiments, the grayscale interpolation method can also be a bilinear interpolation method, a bicubic interpolation method, or other interpolation methods.

[0064] Because the visible light image was converted to grayscale in S12, the fused image obtained in S10 is a grayscale fused image. To further improve the recognizability of the fused image, in some embodiments, such as... Figure 2As shown, after S10, the image fusion method further includes S20: transforming the fused image into a fused pseudo-color image according to a preset pseudo-color mapping table.

[0065] To further clarify the image fusion method provided in this application, Figure 3 Further embodiments of the image fusion method according to this application are provided. In these further embodiments, the image fusion method provided by this application mainly includes steps S01 to S010, and the processing procedures performed on the original infrared image and visible light image according to the image fusion method of this embodiment are as follows: Figure 3 As shown.

[0066] S01: Acquire the original infrared image and the original visible light image respectively.

[0067] S02: Grayscale conversion. Convert the original visible light image into a grayscale image.

[0068] S03: Determine the coordinate mapping relationship. Determine the coordinate mapping relationship between the original infrared image and the corresponding grayscale image of the original visible light image.

[0069] S04: Alignment Mapping. Based on the coordinate mapping relationship, the corresponding grayscale images of the original infrared image and the visible light image are aligned and mapped to the same coordinate system.

[0070] S05: Image segmentation. The aligned and mapped infrared image and the grayscale image are divided into multiple image blocks to obtain corresponding infrared image blocks and visible light image blocks.

[0071] S06: Fourier Transform. Perform Fourier transforms on the infrared image block and the visible light image block respectively to obtain the corresponding frequency domain image block.

[0072] S07: Image patch fusion. Each frequency domain infrared image patch is fused with its corresponding frequency domain visible light image patch to obtain corresponding frequency domain fused image patches.

[0073] S08: Inverse Fourier Transform. Perform inverse Fourier transform on each frequency domain fused image block to obtain the corresponding spatial domain fused image blocks.

[0074] S09: Image patch merging. The various spatially fused image patches are merged in their original order to restore the entire grayscale fused image.

[0075] S010: Pseudo-color mapping. Based on a preset pseudo-color mapping table, the grayscale fused image is mapped to a pseudo-pseudo-color fused image.

[0076] The image fusion method provided in this application decomposes and extracts information from the original infrared and visible light images through two-dimensional Fourier transform, making full use of all information in the infrared and visible light images. The overall process is relatively simple and is beneficial for processing real-time video streams on devices with limited computing power.

[0077] It should be noted that in the image fusion method provided in this application, the order of the above steps is not limited to the order shown in each embodiment. In other embodiments, the order of some steps can be interchanged.

[0078] like Figure 4 As shown, in some embodiments, this application also provides an image fusion device. The image fusion device includes an image segmentation module 11, a Fourier transform module 12, an image fusion module 13, an inverse Fourier transform module 14, and an image merging module 15.

[0079] The image segmentation module 11 is used to divide the registered infrared image and visible light image into multiple image blocks to obtain corresponding infrared image blocks and visible light image blocks; the Fourier transform module 12 is used to perform Fourier transform on each of the infrared image blocks and the visible light image blocks to obtain corresponding frequency domain infrared image blocks and frequency domain visible light image blocks; the image fusion module 13 is used to fuse each of the frequency domain infrared image blocks with its corresponding frequency domain visible light image blocks according to a preset fusion rule to obtain corresponding frequency domain fused image blocks; the inverse Fourier transform module 14 is used to perform inverse Fourier transform on each of the frequency domain fused image blocks to obtain corresponding spatial domain fused image blocks; and the image merging module 15 is used to merge each of the spatial domain fused image blocks to obtain a fused image.

[0080] like Figure 5 As shown, in some embodiments, this application also provides an image fusion apparatus, whose memory 21 stores a processor 13. When the processor 22 executes the computer program instructions stored in the memory 21, it performs the steps of the image fusion method according to any embodiment of this application.

[0081] Furthermore, this application also provides a computer-readable storage medium storing computer program instructions; when executed by a processor, the computer program instructions implement the steps of the detection method provided according to any embodiment of this application.

[0082] The aforementioned processor may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention. The moving target detection device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0083] The aforementioned memory may include high-speed RAM (Random Access Memory) and may also include NVM (Non-Volatile Memory), such as at least one disk storage device.

[0084] In addition, this application also provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the steps of the ranging method according to any embodiment provided in this application.

[0085] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image fusion method, characterized in that, include: The registered infrared and visible light images are uniformly divided into multiple image blocks along the row and column directions, respectively, to obtain the corresponding infrared image blocks and visible light image blocks; For regions where pseudo-structures are prone to occur during the fusion process, adjacent infrared image blocks and adjacent visible light image blocks are set to have a preset overlapping area. In the remaining regions, there is no overlapping area between any adjacent infrared image blocks and any adjacent visible light image blocks. Perform Fourier transform on each of the infrared image blocks and the visible light image blocks to obtain the corresponding frequency domain infrared image blocks and frequency domain visible light image blocks; Each frequency domain infrared image block is compared with the pixel value at the same position in its corresponding frequency domain visible light image. The pixel at the corresponding position is determined based on the pixel with the larger pixel value, and the corresponding frequency domain fused image block is obtained. Perform an inverse Fourier transform on each of the frequency domain fused image blocks to obtain the corresponding spatial domain fused image blocks; During the sequential merging of the various spatial fusion image blocks, the pixels at the corresponding positions in the fused image are determined based on the pixels in the non-overlapping areas of the spatial fusion image blocks. The distances from the center pixels of two adjacent spatial fusion image blocks to their corresponding overlapping areas are compared, and the pixels at the corresponding positions in the fused image are determined based on the overlapping area pixels of the spatial fusion image blocks that are closer in distance.

2. The image fusion method according to claim 1, characterized in that, The image fusion method further includes: Acquire infrared and visible light images of the same target in the same scene; Convert the visible light image into a grayscale image; Determine the coordinate mapping relationship between the grayscale images of the infrared image and the visible light image; The infrared image and the visible light image are aligned and mapped to the same coordinate system according to the coordinate mapping relationship to obtain the registered infrared image and the visible light image.

3. The image fusion method according to claim 2, characterized in that, The acquisition of infrared and visible light images of the same target in the same scene includes: acquiring infrared and visible light images of the same target in the same scene, respectively, captured by an infrared camera and a visible light camera. Determining the coordinate mapping relationship between the grayscale images of the infrared image and the visible light image includes: Based on the parameter information and relative geometric position information of the infrared camera and the visible light camera, the coordinate mapping relationship between the grayscale images of the infrared image and the visible light image is determined.

4. The image fusion method according to claim 2, characterized in that, The step of aligning and mapping the infrared image and the visible light image to the same coordinate system according to the coordinate mapping relationship to obtain the registered infrared image and the visible light image includes: The infrared image and the visible light image are aligned and mapped to the same coordinate system according to the mapping relationship, and the pixels corresponding to the non-integer pixel coordinates in the same coordinate system are assigned values ​​based on the gray level interpolation method.

5. The image fusion method according to claim 2, characterized in that, After merging the various spatial fusion image blocks to obtain a fused image, the image fusion method further includes: According to the preset pseudo-color mapping table, the fused image is transformed into a fused pseudo-color image.

6. An image fusion device, characterized in that, Including memory and processor; When the processor executes the computer program instructions stored in the memory, it performs the steps of the image fusion method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions; When the computer program instructions are executed by the processor, they implement the steps of the image fusion method according to any one of claims 1 to 5.

Images