Image fusion implementation method based on NPU

Abstract

The invention provides an NPU-based image fusion implementation method, and the method comprises the steps: carrying out the YUV420 conversion of an original infrared light image and an original visible light image, obtaining a first infrared light image and a first visible light image, carrying out the projection transformation of the first infrared light image, and obtaining a second infrared light image; inputting the first visible light image and the second infrared light image into a neural network processing unit to obtain a basic visible light image, a basic infrared light image, a detail visible light image and a detail infrared light image, and fusing the basic infrared light image and the basic visible light image to obtain a basic fusion image; the detail visible light image and the detail infrared light image are fused to obtain a detail fusion image, and the basic fusion image and the detail fusion image are fused to obtain a gray fusion image, compared with the prior art, the image fusion algorithm is achieved through the NPU, the image processing speed is increased, the performance requirement of the algorithm for a CPU is reduced, and the CPU occupancy rate of the algorithm is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of image fusion and image processing, and in particular to a method for realizing image fusion based on NPU. Background Art

[0002] Image fusion technology refers to the process of processing the images of multiple sensors or multiple images of a single sensor of the same target or the same scene according to certain rules to generate a new image with more comprehensive information, including redundant information and complementary information of multiple images. The fusion of infrared and visible light images is an important branch in the field of image fusion. Infrared imaging has long detection distance and strong environmental adaptability, but the details are not rich enough, the structural texture features are not obvious, and there is a lack of color information; visible light images are rich in details, textures and color features, have strong semantics, poor environmental adaptability, and insufficient night vision. The two are complementary.

[0003] Traditional image fusion methods mainly rely on general-purpose central processing units (CPUs) for calculation and processing. However, with the continuous improvement of image resolution and the increasing complexity of processing tasks, image fusion based solely on CPUs faces many challenges. Summary of the invention

[0004] In view of the above defects or deficiencies in the prior art, the present application aims to provide an NPU-based image fusion implementation method, comprising the following steps:

[0005] Acquire an original infrared image and an original visible light image;

[0006] Converting the original infrared light image and the original visible light image into a first infrared light image and a first visible light image, wherein the first infrared light image and the first visible light image are in YUV420 format;

[0007] Performing a projective transformation on the Y channel of the first infrared light image to obtain a second infrared light image spatially aligned with the first visible light image; wherein in the YUV color space, a Y channel, a U channel and a V channel are included, the Y channel is used to store brightness information, and the U channel and the V channel are used to store chromaticity information;

[0008] Inputting the second infrared light image into a neural network processing unit to obtain a basic infrared light image, and inputting the first visible light image into the neural network processing unit to obtain a basic visible light image;

[0009] Obtaining a detailed infrared light image according to the original infrared light image and the basic infrared light image, and obtaining a detailed visible light image according to the original visible light image and the basic visible light image;

[0010] Fusing the basic visible light image and the basic infrared light image to obtain a basic fused image;

[0011] fusing the detail visible light image and the detail infrared light image to obtain a detail fused image;

[0012] The basic fused image and the detail fused image are fused to obtain a grayscale fused image.

[0013] According to the technical solution provided in the embodiment of the present application, the neural network processing unit performs a convolution operation on the second infrared light image and the first visible light image to obtain the basic infrared light image and the basic visible light image. The formula of the convolution operation is shown as follows:

[0014]

[0015] In the formula, B ir1 represents the second infrared image, B iv1 represents the first visible light image; (i, j) represents the position coordinates of the pixel point in the image, and (m, n) represents the offset position coordinates of the filter relative to the image.

[0016] According to the technical solution provided in the embodiment of the present application, after obtaining the grayscale fused image, the following steps are also included:

[0017] The grayscale fused image is color mapped to obtain a color fused image, wherein the format of the color fused image is RGB.

[0018] According to the technical solution provided in the embodiment of the present application, the grayscale fused image is subjected to color mapping to obtain a color fused image, comprising the following steps:

[0019] The original infrared light image is mapped to the R channel, the original visible light image is mapped to the B channel, and the grayscale fused image is mapped to the G channel to obtain the color fused image.

[0020] According to the technical solution provided in the embodiment of the present application, the grayscale fused image is subjected to color mapping to obtain a color mapped fused image, comprising the following steps:

[0021] Replacing the Y channel in the first visible light image with the grayscale fused image to form a color fused image;

[0022] The color fused image is converted into RGB format to obtain the color mapping fused image.

[0023] According to the technical solution provided in the embodiment of the present application, the original infrared light image and the basic infrared light image are subjected to an image difference operation to obtain the detailed infrared light image, and the original visible light image and the basic visible light image are subjected to an image difference operation to obtain the detailed visible light image.

[0024] According to the technical solution provided in the embodiment of the present application, the basic visible light image and the basic infrared light image are fused by a weighted method to obtain the basic fused image, wherein the weights corresponding to the basic visible light image and the basic infrared light image are fixed values.

[0025] According to the technical solution provided in the embodiment of the present application, the detail visible light image and the detail infrared light image are fused by a weighted method to obtain the detail fused image, wherein the weights corresponding to the detail visible light image and the detail infrared light image are determined according to the detail intensities of the detail visible light image and the detail infrared light image, respectively.

[0026] According to the technical solution provided in the embodiment of the present application, the basic fused image and the detail fused image are fused by using a superposition method to obtain the grayscale fused image.

[0027] In summary, the present application proposes an image fusion implementation method based on NPU, which converts the original infrared image and the original visible light image into a first infrared image and a first visible light image through YUV420 conversion, and then projects the Y channel of the first infrared image to obtain a second infrared image spatially aligned with the first visible light image, and the first visible light image and the second infrared image are input into a neural network processing unit to obtain a basic visible light image and a basic infrared image, and then obtain a detailed visible light image and a detailed infrared image based on the original visible light image and the original infrared image, and then fuse the basic infrared image and the basic visible light image to obtain a basic fused image, fuse the detailed visible light image and the detailed infrared image to obtain a detailed fused image, and finally fuse the basic fused image and the detailed fused image to obtain a grayscale fused image. Compared with the prior art, the present application implements an image fusion algorithm through NPU, speeds up image processing, reduces the performance requirements of the algorithm on the CPU, and reduces the CPU occupancy of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a method for implementing image fusion based on NPU provided in an embodiment of the present application;

[0029] Figure 2 A schematic diagram of an original infrared image provided in an embodiment of the present application;

[0030] Figure 3A schematic diagram of an original visible light image provided in an embodiment of the present application;

[0031] Figure 4 A schematic diagram of a color fused image provided in an embodiment of the present application; DETAILED DESCRIPTION

[0032] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0033] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0034] As mentioned in the background technology, this application proposes a method for implementing image fusion based on NPU, including the following steps:

[0035] S100. Acquire an original infrared image and an original visible light image;

[0036] Optionally, the original infrared image and the original visible light image can be acquired by a visible light camera and an infrared thermal imager; or the original infrared image or the original visible light image can be acquired by reading the original infrared image or the original visible light image from a storage device. The method of acquiring the image is not limited. Figure 2 As shown, the original visible light image is as Figure 3 shown.

[0037] S200. Converting the original infrared light image and the original visible light image into a first infrared light image and a first visible light image, wherein the first infrared light image and the first visible light image are in YUV420 format;

[0038] Among them, the brightness information and chromaticity information of the original infrared light image and the original visible light image are stored separately according to the specific rules of YUV420, and the chromaticity is downsampled to reduce the amount of data to obtain the first infrared light image and the first visible light image. The YUV420 conversion is the existing technology and will not be repeated here.

[0039] S300. Performing a projection transformation on the Y channel of the first infrared light image to obtain a second infrared light image spatially aligned with the first visible light image; wherein, in the YUV color space, it includes a Y channel, a U channel and a V channel, the Y channel is used to store brightness information, and the U channel and the V channel are used to store chromaticity information;

[0040] The purpose of performing the projection transformation on the Y channel of the first infrared light image is to adjust the viewing angle of the first infrared light image so that it matches the viewing angle of the first visible light image as much as possible. Changing the position and direction of the pixels in the first infrared light image through specific mathematical transformation to achieve spatial alignment with the first visible light image is a prior art and will not be described in detail here.

[0041] S400. Inputting the second infrared light image into a neural network processing unit to obtain a basic infrared light image, and inputting the first visible light image into the neural network processing unit to obtain a basic visible light image;

[0042] Among them, NPU (Neural Processing Unit) is a processor specially designed to accelerate artificial intelligence tasks. The neural network processing unit performs a convolution operation on the second infrared light image and the first visible light image to obtain the basic infrared light image and the basic visible light image. The formula of the convolution operation is shown as follows:

[0043]

[0044] In the formula, B ir1 represents the second infrared image, B iv1 represents the first visible light image; (i, j) represents the position coordinates of the pixel point in the image, and (m, n) represents the offset position coordinates of the filter relative to the image.

[0045] The original infrared image and the original visible light image both include high-frequency image information and low-frequency image information. After the convolution operation, the high-frequency image information is filtered out and the low-frequency image information is retained, which is the basic infrared image or the basic visible light image. Optionally, the number of convolution operations in this solution is 3.

[0046] S500. Obtaining a detailed infrared image according to the original infrared image and the basic infrared image, and obtaining a detailed visible light image according to the original visible light image and the basic visible light image;

[0047] In a preferred embodiment, the original infrared light image and the basic infrared light image are subjected to an image difference operation to obtain the detailed infrared light image, and the original visible light image and the basic visible light image are subjected to an image difference operation to obtain the detailed visible light image; since the original visible light image and the original infrared light image both include high-frequency image information and low-frequency image information, and the basic infrared light image and the basic visible light image after decomposition are both low-frequency image information, high-frequency image information can be obtained by difference, namely the detailed visible light image and the detailed infrared light image.

[0048] The pixel values ​​of the detailed infrared light image are obtained by traversing each pixel of the original infrared light image and the basic infrared light image, and the pixel values ​​of the corresponding positions are subtracted, thereby obtaining the pixel values ​​of the detailed infrared light image. Similarly, the pixel values ​​of the detailed visible light image are obtained by traversing each pixel of the original visible light image and the basic visible light image, and the pixel values ​​of the corresponding positions are subtracted, thereby obtaining the detailed visible light image.

[0049] S600. Fusing the basic visible light image and the basic infrared light image to obtain a basic fused image;

[0050] In a preferred embodiment, the basic visible light image and the basic infrared light image are fused by a weighted method to obtain the basic fused image, wherein the weights corresponding to the basic visible light image and the basic infrared light image are fixed values.

[0051] The fusion of the basic visible light image and the basic infrared light image is achieved by the following formula:

[0052] B fusion (x,y)=p biv ×B iv (x,y)+p bir ×B ir (x,y)

[0053] p biv =Avg(B iv ) / (Avg(B iv )+Avg(B ir )) Formula (1)

[0054] p bir =Avg(B ir ) / (Avg(B iv )+Avg(B ir ))

[0055] In the formula, B ir represents the basic infrared image, Biv represents the basic visible light image; p biv represents the basic visible light image fusion coefficient, p bir Represents the basic infrared image fusion coefficient; B fusion represents the basic fusion image, x represents the horizontal position of the image pixel, y represents the vertical position of the image pixel, Avg(B iv ) represents the average value of the basic visible light image, Avg(B ir ) represents the mean value of the infrared visible light image.

[0056] S700. Fusing the detail visible light image and the detail infrared light image to obtain a detail fused image;

[0057] In a preferred embodiment, the detail visible light image and the detail infrared light image are fused by a weighted method to obtain the detail fused image, wherein the weights corresponding to the detail visible light image and the detail infrared light image are determined according to the detail intensities of the detail visible light image and the detail infrared light image, respectively.

[0058] The fusion of the detailed visible light image and the detailed infrared light image is achieved by the following formula:

[0059] D fusion (x,y)=p div (x,y)×D iv (x,y)+p dir (x,y)×D ir (x,y)

[0060] p div (x,y)=D iv (x,y) / (D iv (x,y)+D ir (x,y)) Formula (2)

[0061] p dir (x,y)=D ir (x,y) / (D iv (x,y)+D ir (x,y)

[0062] Where D fusion represents the detail fusion image, D iv Represents the detailed visible light image, D ir Represents the detailed infrared image.

[0063] S800. Fusing the basic fused image and the detail fused image to obtain a grayscale fused image;

[0064] In a preferred embodiment, the basic fused image and the detail fused image are fused by using a superposition method to obtain the grayscale fused image.

[0065] The grayscale fused image is obtained by summing the pixel values ​​at corresponding positions in the basic fused image and the detail fused image.

[0066] In a preferred embodiment, after obtaining the grayscale fused image, the following steps are also included:

[0067] S900. Perform color mapping on the grayscale fused image to obtain a color fused image, wherein the format of the color fused image is RGB. The grayscale fused image is not suitable for human eye observation, and the grayscale fused image can be color mapped by the following two different methods; wherein, in color images, the RGB (Red Green Blue) color mode is usually used, where the R channel is the red channel, the B channel is the blue channel, and the G channel is the green channel.

[0068] In a preferred embodiment, step S900 includes the following steps:

[0069] S910. Map the original infrared image to the R channel, map the original visible light image to the B channel, and map the grayscale fused image to the G channel to obtain the color fused image, such as Figure 4 shown.

[0070] In a preferred embodiment, step S900 includes the following steps:

[0071] S920. Replace the Y channel in the first visible light image with the grayscale fused image to form a color fused image;

[0072] S921. Convert the color fused image into RGB format to obtain the color mapping fused image.

[0073] In summary, this application implements a multi-scale image fusion algorithm through NPU, speeds up image processing, reduces the algorithm's performance requirements on the CPU, reduces the algorithm's CPU occupancy, and also uses a color mapping method to color the fused image to improve the visual effect of the fused image.

[0074] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

Claims

1. A method for realizing image fusion based on NPU, characterized in that: The steps include: Acquire an original infrared image and an original visible light image; Converting the original infrared light image and the original visible light image into a first infrared light image and a first visible light image, wherein the first infrared light image and the first visible light image are in YUV420 format; Performing a projective transformation on the Y channel of the first infrared light image to obtain a second infrared light image spatially aligned with the first visible light image; wherein in the YUV color space, a Y channel, a U channel and a V channel are included, the Y channel is used to store brightness information, and the U channel and the V channel are used to store chromaticity information; Inputting the second infrared light image into a neural network processing unit to obtain a basic infrared light image, and inputting the first visible light image into the neural network processing unit to obtain a basic visible light image; Obtaining a detailed infrared light image according to the original infrared light image and the basic infrared light image, and obtaining a detailed visible light image according to the original visible light image and the basic visible light image; Fusing the basic visible light image and the basic infrared light image to obtain a basic fused image; fusing the detail visible light image and the detail infrared light image to obtain a detail fused image; The basic fused image and the detail fused image are fused to obtain a grayscale fused image.

2. The NPU-based image fusion implementation method according to claim 1, characterized in that: The neural network processing unit performs a convolution operation on the second infrared light image and the first visible light image to obtain the basic infrared light image and the basic visible light image. The formula of the convolution operation is shown as follows: In the formula, B ir1 represents the second infrared image, B iv1 represents the first visible light image; (i, j) represents the position coordinates of the pixel in the image, and (m, n) represents the offset position coordinates of the filter relative to the image.

3. The NPU-based image fusion implementation method according to claim 1, characterized in that: After obtaining the grayscale fusion image, the following steps are also included: The grayscale fused image is color mapped to obtain a color fused image, wherein the format of the color fused image is RGB.

4. The NPU-based image fusion implementation method according to claim 3, characterized in that: The step of color mapping the grayscale fused image to obtain a color fused image comprises the following steps: The original infrared light image is mapped to the R channel, the original visible light image is mapped to the B channel, and the grayscale fused image is mapped to the G channel to obtain the color fused image.

5. The NPU-based image fusion implementation method according to claim 3, characterized in that: The step of color mapping the grayscale fused image to obtain a color mapped fused image comprises the following steps: Replacing the Y channel in the first visible light image with the grayscale fused image to form a color fused image; The color fused image is converted into RGB format to obtain the color mapping fused image.

6. The NPU-based image fusion implementation method according to claim 1, characterized in that: The original infrared light image and the basic infrared light image are subjected to an image difference operation to obtain the detailed infrared light image, and the original visible light image and the basic visible light image are subjected to an image difference operation to obtain the detailed visible light image.

7. The NPU-based image fusion implementation method according to claim 1, characterized in that: The basic visible light image and the basic infrared light image are fused by a weighted method to obtain the basic fused image, wherein the weights corresponding to the basic visible light image and the basic infrared light image are fixed values.

8. The method for realizing image fusion of gene NPU according to claim 1, characterized in that: The detail visible light image and the detail infrared light image are fused by a weighted method to obtain the detail fused image, wherein the weights corresponding to the detail visible light image and the detail infrared light image are determined according to the detail intensities of the detail visible light image and the detail infrared light image, respectively.

9. The NPU-based image fusion implementation method according to claim 1, characterized in that: The basic fused image and the detail fused image are fused by using a superposition method to obtain the grayscale fused image.

Images