Image processing method, image processing device and computer readable storage medium

Abstract

The invention discloses an image processing method and device and a computer readable storage medium, and the method comprises the steps: carrying out the N-layer pyramid decomposition of a visible light original image, and obtaining a visible light basic layer; performing N-layer pyramid decomposition on the infrared original image to obtain an infrared base layer; and according to the infrared pixel weight curve and the visible light pixel weight curve, carrying out image fusion on the visible light base layer and the infrared base layer to obtain a base layer fusion image. N-layer pyramid decomposition is carried out on a visible light original image to obtain a visible light base layer; n-layer pyramid decomposition is carried out on the original infrared image to obtain an infrared base layer, and the visible light base layer and the infrared base layer are fused according to the infrared pixel weight curve and the visible light pixel weight curve, so that the display weights of different pixel values of the infrared base layer in the base layer fused image are high and low; and the highlight degree of the fused infrared target can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image acquisition, and in particular to an image processing method and an image processing device thereof, and a computer-readable storage medium. Background Art

[0002] Infrared imaging is not affected by the intensity of ambient light and can work in low-light environments. Therefore, infrared images are widely used in target recognition and detection, but their texture details are relatively fuzzy; visible light images can provide rich texture details. After fusing visible light images and infrared images, the fused image obtained can have the advantages of both.

[0003] However, the fusion effect of visible light images and infrared images in the prior art is not good, for example, the infrared target is not prominent enough, the image lacks naturalness, and the noise is large. Summary of the invention

[0004] The technical problem solved by the present invention is how to improve the fusion effect of visible light images and infrared images.

[0005] To solve the above technical problems, an embodiment of the present invention provides an image processing method, including: performing N-layer pyramid decomposition on a visible light original image to obtain a visible light base layer; performing N-layer pyramid decomposition on an infrared original image to obtain an infrared base layer, wherein N is a natural number; and performing image fusion on the visible light base layer and the infrared base layer according to an infrared pixel weight curve and a visible light pixel weight curve to obtain a base layer fused image.

[0006] Optionally, the image processing method also includes: setting an infrared pixel weight curve; the step of setting the infrared pixel weight curve includes: setting a background target area, a hot target area and a cold target area within the infrared base layer pixel value range, wherein the infrared pixel value in the background target area is between the infrared pixel value in the hot target area and the infrared pixel value in the cold target area; setting an infrared pixel weight curve for the background target area, an infrared pixel weight curve for the hot target area and an infrared pixel weight curve for the cold target area respectively, wherein the infrared pixel weight corresponding to the infrared pixel value in the background target area is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the hot target area; the infrared pixel weight corresponding to the infrared pixel value in the background target area is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the cold target area.

[0007] Optionally, in the step of setting a background target area, a hot target area and a cold target area within the infrared base layer pixel value range, an average value of the infrared base layer pixel values ​​is located within the background target area.

[0008] Optionally, an initial infrared pixel weight corresponding to the minimum infrared pixel value, an initial infrared pixel weight corresponding to the first infrared pixel value, an initial infrared pixel weight corresponding to the second infrared pixel value, and an initial infrared pixel weight corresponding to the maximum infrared pixel value are set respectively, wherein the minimum infrared pixel value, the first infrared pixel value, the second infrared pixel value, and the maximum infrared pixel value increase in sequence, the initial infrared pixel weight corresponding to the minimum infrared pixel value is greater than or equal to the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the maximum infrared pixel value is greater than or equal to the initial infrared pixel weight corresponding to the second infrared pixel value, and the area between the minimum infrared pixel value and the first infrared pixel value is the cold target area, The area between the first infrared pixel value and the second infrared pixel value is the background target area, and the area between the second infrared pixel value and the maximum infrared pixel value is the hot target area; the infrared pixel weight curve of the cold target area is obtained according to the initial infrared pixel weight corresponding to the minimum infrared pixel value and the initial infrared pixel weight corresponding to the first infrared pixel value; the infrared pixel weight curve of the background target area is obtained according to the initial infrared pixel weight corresponding to the first infrared pixel value and the initial infrared pixel weight corresponding to the second infrared pixel value; the infrared pixel weight curve of the hot target area is obtained according to the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value.

[0009] Optionally, the step of obtaining a base layer fused image includes: calculating a base layer fused image Gnr = w_vis*Gnv + w_ir*Gni according to the infrared pixel weight curve, the visible light pixel weight curve, each visible light pixel value in the visible light base layer and each infrared pixel value in the infrared base layer, where Gnr represents the base layer fused image, w_vis represents the visible light pixel weight, w_ir represents the infrared pixel weight, Gnv represents the visible light pixel value, Gni represents the infrared pixel value, and n∈N, where n is a natural number.

[0010] Optionally, performing N-layer pyramid decomposition on the visible light original image also obtains N layers of visible light detail layers; performing N-layer pyramid decomposition on the infrared original image also obtains N layers of infrared detail layers; wherein N is greater than or equal to 1; the image processing method further includes: performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image; wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1.

[0011] Optionally, in the step of performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image, the ath visible light detail layer and the ath infrared detail layer are averagely fused to obtain the ath detail fused image.

[0012] Optionally, the step of performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image includes: performing image fusion on the ath visible light detail layer and the ath infrared detail layer according to the ath detail fusion weight to obtain the ath detail fused image.

[0013] Optionally, the step of obtaining the ath detail fusion image includes: calculating the detail layer fusion image Lar=Lav*weight according to the ath detail fusion weight, each infrared pixel value in the infrared detail layer, and each visible light pixel value in the visible light detail layer. Lavi +Lai*(1-weight Lavi ), Lar represents the fused image of the ath detail layer, weight Lavi represents the ath detail fusion weight, Lav is the ath visible light detail layer, and Lai is the ath infrared detail layer.

[0014] Optionally, the step of performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image further includes: obtaining the ath detail fusion weight.

[0015] Optionally, the step of obtaining the ath detail fusion weight includes: obtaining the activity of the ath infrared detail layer based on the ath infrared detail layer; obtaining the activity of the ath visible light detail layer based on the ath visible light detail layer; obtaining the ath layer detail fusion weight based on the activity of the ath infrared detail layer and the activity of the ath visible light detail layer, combined with the ath infrared noise reduction parameter and the ath visible light noise reduction parameter.

[0016] Optionally, the method further includes: performing image fusion on the ath visible light detail layer and the ath infrared detail layer, and before the step of obtaining the ath detail fused image: performing pinch-off operations on the ath visible light detail layer and the ath infrared detail layer respectively; in the step of obtaining the ath detail fused image, performing image fusion on the ath visible light detail layer and the ath infrared detail layer that have been pinched off, to obtain the ath detail fused image.

[0017] Optionally, the image processing method further includes: performing image restoration according to the base layer fusion image to obtain a grayscale fusion image.

[0018] Optionally, N layers of pyramid decomposition are performed on the visible light original image to obtain N layers of visible light detail layers; N layers of pyramid decomposition are performed on the infrared original image to obtain N layers of infrared detail layers; wherein N is greater than or equal to 1; the image processing method also includes: performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image; wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1; in the step of obtaining a grayscale fused image, image restoration is performed based on the base layer fused image, the 0th detail fused image, ..., the ath detail fused image, ..., the N-1th detail fused image to obtain a grayscale fused image.

[0019] Optionally, in at least one of the steps of performing N-layer pyramid decomposition on the visible light original image and performing N-layer pyramid decomposition on the infrared original image, N-layer Laplacian pyramid decomposition is performed on the image to be processed; wherein the image to be processed is one of the visible light original image and the infrared original image.

[0020] Optionally, the step of performing N-layer Laplacian pyramid decomposition includes: executing the Laplacian pyramid decomposition step at least once, executing the Laplacian pyramid decomposition step for the jth time to obtain the j-1th Laplacian layer, where j is a positive integer greater than or equal to 0 and less than or equal to N; wherein the m-th execution of the Laplacian pyramid decomposition step includes: performing Gaussian filtering on the m-1th layer image; performing 1 / 2 down-sampling on the Gaussian filtered m-1th layer image to obtain the m-1th Gaussian layer; and obtaining the m-1th Laplacian layer according to the m-1th Gaussian layer and the m-1th layer image, wherein m is a positive integer greater than or equal to 2 and less than or equal to N.

[0021] Optionally, in the step of performing Laplacian pyramid decomposition for the first time, Gaussian filtering is performed on the image to be processed to obtain a 0th Laplacian layer.

[0022] Optionally, the image processing method also includes: performing image restoration based on the base layer fusion image to obtain a grayscale fusion image; obtaining two chromaticity layers based on the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer; upsampling the two chromaticity layers m-1 times respectively to obtain two chromaticity channel images; obtaining a color image based on the two chromaticity channel images and the grayscale fusion image.

[0023] Optionally, in the step of obtaining two chromaticity layers according to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer, two chromaticity layers are obtained according to the N-1th visible Gaussian layer and the N-1th infrared Gaussian layer.

[0024] Correspondingly, the present invention also provides an image processing device, comprising: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the processor executes the steps of the above-mentioned image processing method.

[0025] Correspondingly, the present invention also provides a computer-readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, on which computer instructions are stored, and when the computer instructions are executed, the steps of the image processing method described above are executed.

[0026] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0027] In the technical solution of the present invention, the visible light base layer is obtained by performing N-layer pyramid decomposition on the visible light original image; the infrared original image is decomposed into N layers of pyramid to obtain the infrared base layer, and the visible light base layer and the infrared base layer are fused according to the infrared pixel weight curve and the visible light pixel weight curve, so that the display weights of different pixel values ​​of the infrared base layer in the base layer fused image are high and low, thereby achieving the effect of "injecting" the infrared base layer into the visible light base layer, which can effectively improve the prominence of the infrared target after fusion; and because the visible light base layer and the infrared base layer only include structural information, the prominent display of the infrared base layer in the base layer fused image can effectively avoid the introduction of noise information, and can effectively ensure the fusion effect of the visible light base layer and the infrared base layer.

[0028] Furthermore, by setting the background target area, the hot target area and the cold target area within the pixel value range of the infrared base layer, and the infrared pixel weights corresponding to the infrared pixel values ​​in different target areas are different, the display weights of the infrared pixel values ​​in different target areas in the base layer fused image are high and low, thereby ensuring that the infrared base layer can be highlighted in the base layer fused image; and the infrared pixel weights corresponding to the infrared pixel values ​​in the cold target area and the infrared pixel weights corresponding to the infrared pixel values ​​in the hot target area are both greater than the infrared pixel weights corresponding to the infrared pixel values ​​in the background target area, which can effectively improve the contrast and naturalness of the fused image and effectively improve the fusion effect.

[0029] Furthermore, the present invention sets an initial weight value corresponding to the minimum infrared pixel value, an initial infrared pixel weight corresponding to the first infrared pixel value, an initial infrared pixel weight corresponding to the second infrared pixel value, and an initial infrared pixel weight corresponding to the maximum infrared pixel value, that is, sets different initial infrared pixel weights, and obtains infrared pixel weight curves in different target areas according to different initial infrared pixel weights. Subsequently, different infrared weight curves can be obtained by adjusting the infrared pixel weights corresponding to different infrared pixel values, thereby optimizing the infrared weight curve, so that the infrared base layer can be accurately highlighted in the base layer fused image.

[0030] Furthermore, the technical solution of the present invention performs N-layer image decomposition on the visible light original image and the infrared original image to obtain a visible light detail layer and an infrared detail layer, and performs image fusion on the ath visible light detail layer and the ath infrared detail layer according to the ath detail fusion weight. Since the visible light detail layer and the infrared detail layer include noise information, and the calculation of the detail fusion weight can adjust the noise in the visible light detail layer and the infrared detail layer, using the detail fusion weight for image fusion can reduce the noise in the visible light detail layer and the infrared detail layer, thereby ensuring the fusion effect of the visible light detail layer and the infrared detail layer.

[0031] Furthermore, the technical solution of the present invention performs a pinch-off operation on the infrared detail layer and the visible light detail layer, that is, performs threshold limiting processing on the infrared detail layer and the visible light detail layer, so that the pixel values ​​in the infrared detail layer and the visible light detail layer are limited, thereby reducing resource usage and suppressing the generation of halo effect.

[0032] Furthermore, the technical solution of the present invention obtains two chromaticity channel images according to the visible Gaussian layer and the infrared Gaussian layer obtained after decomposing the N-layer pyramid, and performs color component processing according to the two chromaticity channel images and the grayscale fused image to obtain a color output image, thereby expanding the application scenarios of image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flowchart of an embodiment of an image processing method of the present invention;

[0034] Figure 2 1 is a flow chart of steps for performing N-layer pyramid decomposition in an embodiment of an image processing method of the present invention;

[0035] Figure 3 1 is a schematic diagram of an algorithm for performing N-layer pyramid decomposition in an embodiment of an image processing method of the present invention;

[0036] Figure 41 is a simulation schematic diagram of performing N-layer pyramid decomposition in an embodiment of an image processing method of the present invention;

[0037] Figure 5 is a schematic diagram of an infrared pixel weight curve in an embodiment of the image processing method of the present invention;

[0038] Figure 6 is a flow chart of steps for performing detail layer fusion in an embodiment of an image processing method of the present invention;

[0039] Figure 7 is a flowchart of steps for obtaining detail fusion weights in an embodiment of an image processing method of the present invention;

[0040] Figure 8 is a schematic diagram of an image restoration process in an embodiment of an image processing method of the present invention;

[0041] Fig. 9 is a schematic diagram comparing the effects of fused images obtained by using different fusion methods in an embodiment of the image processing method of the present invention;

[0042] Fig.10 is a schematic diagram comparing the effects of fused images obtained by using different fusion methods on the base layer in an embodiment of the image processing method of the present invention;

[0043] Fig.11 is a flow chart of steps for obtaining colorization processing in an embodiment of an image processing method of the present invention;

[0044] Fig.12 is a simulation schematic diagram after colorization processing in an embodiment of the image processing method of the present invention;

[0045] Fig.13 is a schematic flow chart of a pinch-off operation in an embodiment of an image processing method of the present invention;

[0046] Fig.14 is a simulation schematic diagram after the pinch-off operation is processed in an embodiment of the image processing method of the present invention;

[0047] Fig.15 is a schematic diagram of another infrared pixel weight curve in an embodiment of the image processing method of the present invention;

[0048] Fig.16 It is a simulation schematic diagram of image fusion in an embodiment of the image processing method of the present invention. DETAILED DESCRIPTION

[0049] The current image fusion algorithm has the following disadvantages: 1. When performing image fusion, the image noise is also merged, resulting in a fused image with larger noise; 2. After fusion, the infrared thermal target becomes less prominent; 3. The contrast of the fused image is not as good as the original visible light image, and the naturalness of the fused image is not as good as the original visible light image; 4. After the fused image is colorized, the color lacks a natural feel.

[0050] In order to solve the above technical problems, the present invention provides an image processing method, which performs N-layer pyramid decomposition on the original visible light image to obtain a visible light base layer; performs N-layer pyramid decomposition on the original infrared image to obtain an infrared base layer, wherein N is a natural number; and performs image fusion on the visible light base layer and the infrared base layer according to the infrared pixel weight curve and the visible light pixel weight curve to obtain a base layer fused image.

[0051] The visible light base layer and the infrared base layer are fused according to the infrared pixel weight curve and the visible light pixel weight curve, so that the display weight of the infrared base layer in the base layer fused image varies, thereby ensuring the prominence of the infrared base layer and improving the fusion effect of the visible light base layer and the infrared base layer; and because the visible light base layer and the infrared base layer only include structural information, the introduction of noise information is avoided, thereby improving the fusion effect of the visible light base layer and the infrared base layer.

[0052] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0053] refer to Figure 1 , shows a flow chart of an embodiment of an image processing method of the present invention.

[0054] The image processing method comprises:

[0055] Step S1: performing N-layer pyramid decomposition on the original visible light image to obtain a visible light base layer;

[0056] Step S2: performing N-layer pyramid decomposition on the infrared original image to obtain an infrared base layer, where N is a natural number;

[0057] Step S3: performing image fusion on the visible light base layer and the infrared base layer according to the infrared pixel weight curve and the visible light pixel weight curve to obtain a base layer fused image.

[0058] It should be noted that the visible light original image and the infrared original image are obtained by two different detectors. Specifically, the two images are basically aligned mechanically, and then accurately aligned by operations such as image scaling, stretching, and cropping.

[0059] First, at least one of step S1 and step S2 is executed to perform N-layer pyramid decomposition to obtain a base layer. Specifically, step S1 and step S2 may be executed sequentially; step S2 and step S1 may be executed sequentially; or step S1 and step S2 may be executed simultaneously.

[0060] In some embodiments of the present invention, in the step of performing N-layer pyramid decomposition, N-layer Laplacian pyramid decomposition is performed, that is, in at least one of step S1, the step of performing N-layer pyramid decomposition on the visible light original image and step S2, the step of performing N-layer pyramid decomposition on the infrared original image, N-layer Laplacian pyramid decomposition is performed on the image to be processed, wherein the image to be processed is one of the visible light original image and the infrared original image.

[0061] Specifically, in step S1, in the step of performing N-layer pyramid decomposition on the visible light original image, N-layer Laplacian pyramid decomposition is performed on the visible light original image; in step S2, in the step of performing N-layer pyramid decomposition on the infrared original image, N-layer Laplacian pyramid decomposition is performed on the infrared original image.

[0062] In some exemplary embodiments, the step of performing N-layer Laplacian pyramid decomposition includes: performing the Laplacian pyramid decomposition step at least once, performing the Laplacian pyramid decomposition step for the jth time, and obtaining the j-1th Laplacian layer, where j is a positive integer greater than or equal to 0 and less than or equal to N-1.

[0063] The following is combined with Figure 2 , Attachment Figure 3 and attached Figure 4 The technical solution for performing N-layer Laplacian pyramid decomposition in an embodiment of the image processing method of the present invention is described in detail.

[0064] refer to Figure 2 , shows a schematic flow chart of performing the Laplacian pyramid decomposition step for the mth time in an embodiment of the image processing method of the present invention.

[0065] Specifically, the m-th execution of the Laplacian pyramid decomposition step includes: step S11: performing Gaussian filtering on the m-1th layer image; step S12: performing 1 / 2 down-sampling on the Gaussian filtered m-1th layer image to obtain a sampled filtered m-1th layer image; step S13: performing up-sampling on the m-1th layer image to obtain an m-1th Gaussian layer; step S14: obtaining an m-1th Laplacian layer according to the m-1th Gaussian layer and the m-1th layer image, wherein m is a positive integer greater than or equal to 2 and less than or equal to N-1.

[0066] In some embodiments, in step S11: performing Gaussian filtering on the m-1th layer image, the filter kernel of the Gaussian filtering is a 3×3 filter kernel:

[0067]

[0068] Specifically, the downsampling and Gaussian filtering operations performed before the downsampling are: first perform edge filling and Gaussian filtering, and then perform 1 / 2 downsampling, wherein the edge filling uses a copy edge method to fill one row and one column on each of the four sides of the image; the filter kernel of the Gaussian filter is the same as above, and then when performing 1 / 2 downsampling, the values ​​of the even rows and even columns are retained as the downsampling results (counting starts from row 0 and column 0, row 0 is counted as an even row, and column 0 is counted as an even column).

[0069] Specifically, the upsampling operation method is to insert a row of 0s after each row of the image, and insert a row of 0s after each column of the image, to obtain an image with twice the width and height. Note that odd rows and odd columns are 0 (counting starts from row 0 and column 0, row 0 is counted as an even row, and column 0 is counted as an even column).

[0070] In other embodiments of the present invention, when resources are sufficient, in the step of performing Gaussian filtering on the m-1th layer image, the filter kernel of the Gaussian filtering may also be a 5×5 filter kernel:

[0071]

[0072] In the above schemes, the edge transition of the fused image obtained by using a 5×5 filter kernel is more natural.

[0073] Combined with reference Figure 3 and Figure 4 ,in, Figure 3 A schematic diagram of an algorithm for performing N-layer pyramid decomposition in the image processing method of the present invention is shown; Figure 4 A simulation schematic diagram of performing N-layer pyramid decomposition in the image processing method of the present invention is shown.

[0074] like Figure 3 As shown, in the first Laplacian pyramid decomposition step, Gaussian filtering is performed on the 0th layer image G0. The Gaussian filtered 0th layer image is downsampled by 1 / 2↓ to obtain the sampled filtered 1st layer image G1; the sampled filtered 1st layer image G1 is upsampled↑ to obtain the 0th Gaussian layer G0e; according to the 0th Gaussian layer G0e and the 0th layer image G0, the 0th Laplace layer L0 is obtained.

[0075] Right now, L0=G0-G0e.

[0076] It should be noted that, in some embodiments, the step of performing the Laplacian pyramid decomposition for the first time includes performing Gaussian filtering on the image to be processed to obtain the 0th Laplacian layer, wherein the image to be processed is one of the visible light original image and the infrared original image.

[0077] Specifically, in step S1, during the process of performing N-layer pyramid decomposition on the visible light original image, the first Laplacian pyramid decomposition step includes: Figure 4 Gaussian filtering is performed on the image G0 on the left side of the image. The Gaussian filtered layer 0 image is downsampled by 1 / 2↓ to obtain the sampled filtered layer 1 image G1 (such as Figure 4 The sampled and filtered first layer image G1 is upsampled↑ to obtain the 0th Gaussian layer G0e (as shown in the image G1 on the left side of the figure); the sampled and filtered first layer image G1 is upsampled↑ to obtain the 0th Gaussian layer G0e (as shown in the image G1 on the left side of the figure); Figure 4 According to the 0th Gaussian layer G0e and the 0th layer image G0, the 0th Laplacian layer L0 is obtained (as shown in the image G0e on the left side of the figure); Figure 4 (as shown in image L0 on the left).

[0078] Specifically, in step S2, during the process of performing N-layer pyramid decomposition on the infrared original image, the first Laplacian pyramid decomposition step includes: Figure 4 Gaussian filtering is performed on the image G0 on the right side of the image. The Gaussian filtered layer 0 image is downsampled by 1 / 2↓ to obtain the sampled filtered layer 1 image G1 (such as Figure 4 The sampled and filtered first layer image G1 is upsampled↑ to obtain the 0th Gaussian layer G0e (as shown in the image G1 on the right side of the figure); the sampled and filtered first layer image G1 is upsampled↑ to obtain the 0th Gaussian layer G0e (as shown in the image G1 on the right side of the figure); Figure 4 According to the 0th Gaussian layer G0e and the 0th layer image G0, the 0th Laplacian layer L0 is obtained (as shown in the image G0e on the right side of the figure); Figure 4 (as shown in image L0 on the right).

[0079] Continue to refer Figure 3 , in the second Laplace pyramid decomposition step,

[0080] Perform Gaussian filtering on the first layer image G1 The Gaussian filtered first layer image is downsampled by 1 / 2↓ to obtain the sampled filtered second layer image G2; the sampled filtered second layer image G2 is upsampled↑ to obtain the first Gaussian layer G1e; according to the first Gaussian layer G1e and the first layer image G1, the first Laplace layer L1 is obtained.

[0081] Right now, L1=G1-G1e.

[0082] Specifically, in step S1, during the process of performing N-layer pyramid decomposition on the visible light original image, the second Laplacian pyramid decomposition step includes: performing the first layer image G1 (such as Figure 4 Gaussian filtering is performed on the image G1 on the left side of the image. The Gaussian filtered first layer image is downsampled by 1 / 2↓ to obtain the sampled filtered second layer image G2 (such as Figure 4 The sampled and filtered second layer image G2 is upsampled ↑ to obtain the first Gaussian layer G1e (as shown in the image G2 on the left in the figure); the sampled and filtered second layer image G2 is upsampled ↑ to obtain the first Gaussian layer G1e (as shown in the figure Figure 4 According to the first Gaussian layer G1e and the first layer image G1, the first Laplacian layer L1 is obtained (as shown in the image G1e on the left side of FIG. 1 ). Figure 4 (as shown in image L1 on the left).

[0083] Specifically, in step S2, during the process of performing N-layer pyramid decomposition on the infrared original image, the second Laplacian pyramid decomposition step includes: performing the first layer image G1 (such as Figure 4 Gaussian filtering is performed on the image G1 on the right side of the image. The Gaussian filtered first layer image is downsampled by 1 / 2↓ to obtain the sampled filtered second layer image G2 (such as Figure 4 The sampled and filtered second layer image G2 is upsampled ↑ to obtain the first Gaussian layer G1e (as shown in the image G2 on the right side of the figure); the sampled and filtered second layer image G2 is upsampled ↑ to obtain the first Gaussian layer G1e (as shown in the image G2 on the right side of the figure); Figure 4 According to the first Gaussian layer G1e and the first layer image G1, the first Laplacian layer L1 is obtained (as shown in the image G1e on the right side of the figure); Figure 4 (as shown in image L1 on the right).

[0084] Continue to refer Figure 3 In the third Laplacian pyramid decomposition step, Gaussian filtering is performed on the second layer image G2 The Gaussian filtered second layer image is downsampled by 1 / 2↓ to obtain the sampled filtered third layer image G3; the sampled filtered third layer image G3 is upsampled↑ to obtain the second Gaussian layer G2e; according to the second Gaussian layer G2e and the second layer image G2, the second Laplace layer L2 is obtained.

[0085] Right now, L1=G2-G2e.

[0086] Specifically, in step S1, during the process of performing N-layer pyramid decomposition on the visible light original image, the third Laplacian pyramid decomposition step includes: performing the second layer image G2 (such as Figure 4Gaussian filtering is performed on the image G2 on the left side of the image. The Gaussian filtered second layer image is downsampled by 1 / 2↓ to obtain the sampled filtered third layer image G3 (such as Figure 4 The sampled and filtered third layer image G3 is upsampled ↑ to obtain the second Gaussian layer G2e (as shown in the image G3 on the left in the figure); the sampled and filtered third layer image G3 is upsampled ↑ to obtain the second Gaussian layer G2e (as shown in the figure Figure 4 According to the second Gaussian layer G2e and the second layer image G2, the second Laplacian layer L2 is obtained (as shown in the image G1e on the left side of FIG. 1 ); Figure 4 (as shown in image L2 on the left).

[0087] Specifically, in step S2, during the process of performing N-layer pyramid decomposition on the infrared original image, the third Laplace pyramid decomposition step includes: performing the second layer image G2 (such as Figure 4 Gaussian filtering is performed on the image G2 on the right side of the image. The Gaussian filtered second layer image is downsampled by 1 / 2↓ to obtain the sampled filtered third layer image G3 (such as Figure 4 The sampled and filtered third layer image G3 is upsampled ↑ to obtain the second Gaussian layer G2e (as shown in the image G3 on the right side of the figure); the sampled and filtered third layer image G3 is upsampled ↑ to obtain the second Gaussian layer G2e (as shown in the image G3 on the right side of the figure); Figure 4 According to the second Gaussian layer G2e and the second layer image G2, the second Laplacian layer L2 is obtained (as shown in the image G1e on the right side of the figure); Figure 4 (as shown in image L2 on the right).

[0088] The m-th execution of the Laplacian pyramid decomposition step is as described in the above solution, and so on, and the present invention will not repeat them here.

[0089] In some exemplary embodiments, in the step of performing N-layer pyramid decomposition, 3-layer Laplacian pyramid decomposition is performed; the step of performing 3-layer Laplacian pyramid decomposition includes: performing the Laplacian pyramid decomposition step 3 times.

[0090] After performing the Laplacian pyramid decomposition step three times, a 4-layer Gaussian pyramid and a 3-layer Laplacian pyramid are obtained, wherein the 4-layer Gaussian pyramid includes: a 0th layer image G0, a 1st layer image G1, a 2nd layer image G2 and a 3rd layer image G3, and the 3-layer Laplacian pyramid includes: a 0th Laplacian layer L0, a 1st Laplacian layer L1, and a 2nd Laplacian layer L2.

[0091] like Figure 4As shown, the 0th layer image G0 on the left, the 1st layer image G1 on the left, the 2nd layer image G2 on the left and the 3rd layer image G3 on the left are respectively the visible light original images in the 4-layer Gaussian pyramid obtained by performing 3-layer Laplace pyramid decomposition on the visible light original image and the corresponding sampled filtered 1st layer image G1, sampled filtered 2nd layer image G2 and sampled filtered 3rd layer image G3; the 0th layer image L0 on the left, the 1st layer image L1 on the left and the 2nd layer image L2 on the left are respectively the 0th Laplace layer L0, the 1st Laplace layer L1 and the 2nd Laplace layer L2 in the 3-layer Laplace pyramid obtained by performing 3-layer Laplace pyramid decomposition on the visible light original image.

[0092] like Figure 4 As shown, the 0th layer image G0 on the right, the 1st layer image G1 on the right, the 2nd layer image G2 on the right and the 3rd layer image G3 on the right are respectively the infrared original images in the 4-layer Gaussian pyramid obtained by performing 3-layer Laplace pyramid decomposition on the infrared original image and the corresponding sampled filtered 1st layer image G1, sampled filtered 2nd layer image G2 and sampled filtered 3rd layer image G3; the 0th layer image L0 on the left, the 1st layer image L1 on the left and the 2nd layer image L2 on the left are respectively the 0th Laplace layer L0, the 1st Laplace layer L1 and the 2nd Laplace layer L2 in the 3-layer Laplace pyramid obtained by performing 3-layer Laplace pyramid decomposition on the infrared original image.

[0093] In some embodiments of the present invention, at least one of step S1 and step S2 is executed, and in the step of performing N-layer pyramid decomposition, a detail layer is also obtained.

[0094] The base layer is a thumbnail of the original image. After the N-layer pyramid decomposition, the base layer obtained contains extremely low noise information, and may be almost noise-free. The base layer only carries basic structural information of the original image.

[0095] The detail layer only carries information such as texture, edge, and noise, so the fusion strategy of the base layer is the most critical, which is related to the basic framework, contrast, and prominence of the infrared target of the fused image.

[0096] For details, please refer to Figure 1 , in the step of performing N-layer pyramid decomposition, a base layer is obtained: that is, step S1, performing N-layer pyramid decomposition on the visible light original image to obtain a visible light base layer; step S2, performing N-layer pyramid decomposition on the infrared original image to obtain an infrared base layer.

[0097] For details, please refer to Figure 1In step S1, in the step of performing N-layer pyramid decomposition on the visible light original image, N layers of visible light detail layers are obtained; in step S2, in the step of performing N-layer pyramid decomposition on the infrared original image, N layers of infrared detail layers are obtained.

[0098] In some embodiments of the present invention, the highest layer of the Gaussian pyramid, that is, the sampled and filtered Nth layer image, is the base layer; and each Laplacian layer in the Laplacian pyramid is a detail layer.

[0099] Specifically, the highest layer of a Gaussian pyramid obtained by performing N-layer Laplacian pyramid decomposition on the visible light original image is the visible light base layer; the highest layer of a Gaussian pyramid obtained by performing N-layer Laplacian pyramid decomposition on the infrared original image is the infrared base layer.

[0100] Specifically, the Laplacian pyramid obtained by performing N-layer Laplacian pyramid decomposition on the visible light original image is N-layer visible light detail layer; the Laplacian pyramid obtained by performing N-layer Laplacian pyramid decomposition on the infrared original image is N-layer infrared detail layer.

[0101] For example, the 0th Laplace layer L0, the 1st Laplace layer L1 and the 2nd Laplace layer L2 in the Laplace pyramid obtained by performing N-layer Laplace pyramid decomposition on the visible light original image are the 0th visible light detail layer, the 1st visible light detail layer and the 2nd visible light detail layer respectively; the 0th Laplace layer L0, the 1st Laplace layer L1 and the 2nd Laplace layer L2 in the Laplace pyramid obtained by performing N-layer Laplace pyramid decomposition on the infrared original image are the 0th infrared detail layer, the 1st infrared detail layer and the 2nd infrared detail layer respectively.

[0102] Continue to refer Figure 1 After obtaining the visible light base layer and the infrared light base layer, the image processing method further includes: step S3, performing image fusion on the visible light base layer and the infrared base layer according to the infrared pixel weight curve and the visible light pixel weight curve to obtain a base layer fused image.

[0103] The infrared pixel weight relationship is suitable for characterizing the proportion of infrared pixel values ​​in the obtained base layer fusion image. Based on the infrared pixel weight curve and the visible light pixel weight curve, the visible light base layer and the infrared base layer are fused, so that the display weights of different pixel values ​​of the infrared base layer in the base layer fusion image are high and low, thereby achieving the effect of "injecting" the infrared base layer into the visible light base layer, which can effectively improve the prominence of the infrared target after fusion; and because the visible light base layer and the infrared base layer only include structural information, the prominent display of the infrared base layer in the base layer fusion image can effectively avoid the introduction of noise information, and can effectively ensure the fusion effect of the visible light base layer and the infrared base layer.

[0104] In some embodiments of the present invention, Figure 1 As shown, the image processing method further includes: setting an infrared pixel weight curve.

[0105] For details, please refer to Figure 1 , the steps of setting the infrared pixel weight curve include:

[0106] Step S4: setting a background target area, a hot target area and a cold target area within the infrared base layer pixel value range, wherein the infrared pixel value in the background target area is between the infrared pixel value in the hot target area and the infrared pixel value in the cold target area;

[0107] Step S5: respectively setting the infrared pixel weight curve of the background target field, the infrared pixel weight curve of the hot target field, and the infrared pixel weight curve of the cold target field;

[0108] The infrared pixel weight corresponding to the infrared pixel value in the background target field is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the hot target field; the infrared pixel weight corresponding to the infrared pixel value in the background target field is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the cold target field.

[0109] In some embodiments of the present invention, in step S4, in the step of setting the background target area, the hot target area and the cold target area within the infrared base layer pixel value range, the average value Xmean of the infrared base layer pixel values ​​is located within the background target area.

[0110] The average value Xmean of the pixel value is close to the background brightness, and the average value Xmean of the infrared base layer pixel value is close to the background brightness of the infrared base layer. The background target field, hot target field and cold target field are divided so that the average value Xmean of the infrared base layer pixel value is located in the background target field, and the background target field is located around the background brightness of the infrared base layer, and the hot target field and the cold target field are both deviated from the background brightness; and the method of making the average value Xmean of the infrared base layer pixel value be located in the background target field and dividing the background target field, hot target field and cold target field makes the set infrared pixel weight curve a dynamic curve, and the set infrared pixel weight curve will change with the change of the average value Xmean of the infrared base layer pixel value.

[0111] In some embodiments of the present invention, the infrared base layer pixel value ranges from 0 to 256, the average value of the infrared base layer pixel value is real-time statistics, and when the absolute value of the difference between the current infrared base layer pixel value and the average value of the infrared base layer pixel value is greater than a preset difference threshold, the current infrared base layer pixel value is located in the cold target area or the hot target area. For example, the preset difference threshold is 40.

[0112] When the difference between the current infrared base layer pixel value and the average value of the infrared base layer pixel values ​​is greater than a preset difference threshold, and the difference is a positive number, the current infrared base layer pixel value is located within the thermal target area.

[0113] When the difference between the current infrared base layer pixel value and the average value of the infrared base layer pixel values ​​is greater than a preset difference threshold, and the difference is a negative number, the current infrared base layer pixel value is located in the cold target area.

[0114] refer to Figure 5 , Figure 5 The figure shows a schematic diagram of an infrared pixel weight curve in the image processing method of the present invention. In the figure, the X-axis coordinate is represented by the infrared pixel value IR, and the Y-axis coordinate is represented by the infrared pixel weight weight.

[0115] In some embodiments of the present invention, step S5 of respectively setting the infrared pixel weight curve of the background target field, the infrared pixel weight curve of the hot target field, and the infrared pixel weight curve of the cold target field includes: respectively setting the initial infrared pixel weight W1 corresponding to the minimum infrared pixel value Xmin, the initial infrared pixel weight W2 corresponding to the first infrared pixel value X1, the initial infrared pixel weight W3 corresponding to the second infrared pixel value X2, and the initial infrared pixel weight W4 corresponding to the maximum infrared pixel value Xmax.

[0116] Among them, the minimum infrared pixel value Xmin, the first infrared pixel value X1, the second infrared pixel value X2 and the maximum infrared pixel value Xmax increase in sequence, the initial infrared pixel weight W1 corresponding to the minimum infrared pixel value Xmin is greater than or equal to the initial infrared pixel weight W2 corresponding to the first infrared pixel value X1, the initial infrared pixel weight W4 corresponding to the maximum infrared pixel value Xmax is greater than or equal to the initial infrared pixel weight W3 corresponding to the second infrared pixel value X2, and the area between the minimum infrared pixel value Xmin and the first infrared pixel value X1 is the cold target area, the area between the first infrared pixel value X1 and the second infrared pixel value X2 is the background target area, and the area between the second infrared pixel value X2 and the maximum infrared pixel value Xmax is the hot target area.

[0117] In some embodiments of the present invention, step S5 of respectively setting the infrared pixel weight curve of the background target field, the infrared pixel weight curve of the hot target field, and the infrared pixel weight curve of the cold target field further includes: obtaining the infrared pixel weight curve of the cold target field according to the initial infrared pixel weight corresponding to the minimum infrared pixel value and the initial infrared pixel weight corresponding to the first infrared pixel value; obtaining the infrared pixel weight curve of the background target field according to the initial infrared pixel weight corresponding to the first infrared pixel value and the initial infrared pixel weight corresponding to the second infrared pixel value; obtaining the infrared pixel weight curve of the hot target field according to the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value.

[0118] like Figure 5 As shown, the infrared pixel weight curve in the hot target field and the infrared pixel weight curve in the cold target field are both made greater than the infrared pixel weight curve in the background target field, that is, the weight of the infrared base layer of the target that significantly deviates from the background brightness in the base layer fusion image is increased to appropriately highlight it: the target whose pixel value is significantly greater than the average value Xmean of the infrared base layer pixel value, that is, the "hot target" with a temperature significantly higher than the background temperature, is a more important target in the infrared image (such as a car driving in the woods), which needs to be highlighted and a higher weight is set; the target whose pixel value is significantly less than the average value Xmean of the infrared base layer pixel value, that is, the "cold target" with a temperature significantly lower than the background temperature is also a more important target in the infrared image (such as a vehicle that has been stationary in the woods for a period of time), which also needs to be highlighted and a higher weight is also set.

[0119] Specifically, Figure 5 As shown, the horizontal coordinate of the first endpoint A1 is the minimum infrared pixel value Xmin, and the vertical coordinate of the first endpoint A1 is the initial infrared pixel weight W1 corresponding to the minimum infrared pixel value Xmin; the horizontal coordinate of the second endpoint A2 is the first infrared pixel value X1, and the vertical coordinate of the second endpoint A2 is the initial infrared pixel weight W2 corresponding to the first infrared pixel value X1; the horizontal coordinate of the third endpoint A3 is the second infrared pixel value X2, and the vertical coordinate of the third endpoint A3 is the initial infrared pixel weight W3 corresponding to the second infrared pixel value X2; the horizontal coordinate of the fourth endpoint A4 is the maximum infrared pixel value Xmax, and the vertical coordinate of the fourth endpoint A4 is the initial infrared pixel weight W4 corresponding to the maximum infrared pixel value Xmax.

[0120] Continue to refer Figure 5The infrared pixel weight curve is obtained by connecting the first endpoint A1 and the second endpoint A2 to obtain the infrared pixel weight curve L1 in the cold target field; connecting the second endpoint A2 and the third endpoint A3 to obtain the infrared pixel weight curve L2 in the background target field; connecting the third endpoint A3 and the fourth endpoint A4 to obtain the infrared pixel weight curve L3 in the hot target field.

[0121] like Figure 5 In some of the embodiments shown, the coordinates of the first endpoint A1 are (0.3, 0), the coordinates of the second endpoint A2 are (0.1, xmean-dx1), the coordinates of the third endpoint A3 are (0, xmean+dx2), and the coordinates of the fourth endpoint A4 are (0.9, 255), wherein dx1 is the distance between xmean and the horizontal coordinate of the second endpoint A2, and dx2 is the distance between xmean and the horizontal coordinate of the third endpoint A3.

[0122] Continue to refer Figure 4 After obtaining the infrared pixel weight curve, according to the infrared pixel weights corresponding to each infrared pixel in the infrared pixel weight curve, each infrared pixel in the infrared base layer is "injected" into the visible light base layer according to the corresponding infrared pixel weight, which can effectively improve the prominence of the infrared target after fusion, and can effectively improve the contrast and naturalness of the fused image, wherein the image after the fusion of the infrared base layer and the visible light base layer is F3.

[0123] In the above scheme, by setting the initial weight value corresponding to the minimum infrared pixel value, the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value, that is, setting different initial infrared pixel weights, and according to different initial infrared pixel weights, obtaining infrared pixel weight curves in different target areas, subsequently by adjusting the infrared pixel weights corresponding to different infrared pixel values, different infrared weight curves can be obtained, thereby optimizing the infrared weight curve, which can effectively improve the prominence of the infrared target after fusion.

[0124] Continue to refer Figure 1 In some embodiments of the present invention, the image processing method further includes: Step S6: setting a visible light pixel weight curve.

[0125] In some embodiments of the present invention, for the same pixel value, the sum of the visible light pixel weight and the infrared pixel weight is 1; in step S6 of setting the visible light pixel weight curve, the visible light pixel weight curve is set according to the infrared pixel weight curve.

[0126] In some exemplary embodiments, the weight of each visible light pixel in the visible light pixel weight curve is 1-w_ir, where w_ir is the weight of each infrared pixel.

[0127] Continue to refer Figure 1 In some embodiments, in step S3, in the step of performing image fusion on the visible light base layer and the infrared base layer to obtain a base layer fused image, the base layer fused image Gnr is calculated according to the infrared pixel weight curve, the visible light pixel weight curve, the visible light pixel value in the visible light base layer and the infrared pixel value in the infrared base layer: Gnr = w_vis*Gnv+w_ir*Gni, Gnr represents the base layer fused image, w_vis represents the visible light pixel weight, w_ir represents the infrared pixel weight, Gnv represents the visible light pixel value, Gni represents the infrared pixel value, n∈N, and n is a natural number.

[0128] like Figure 1 As shown, in some embodiments of the present invention, step S1, performs N-layer pyramid decomposition on the visible light original image, and obtains N-layer visible light detail layers, wherein the N-layer visible light detail layers include: the 0th visible light detail layer, the 1st visible light detail layer, ..., the N-1th visible light detail layer; step S2, performs N-layer pyramid decomposition on the infrared original image, and obtains N-layer infrared detail layers, wherein the N-layer infrared detail layers include: the 0th infrared detail layer, the 1st infrared detail layer, ..., the N-1th infrared detail layer.

[0129] like Figure 4 In some of the illustrated embodiments, N=3; the three visible light detail layers include: a 0th visible light detail layer, a 1st visible light detail layer, and a 2nd visible light detail layer, which are respectively the 0th Laplace layer L0, the 1st Laplace layer L1, and the 2nd Laplace layer L2 in the Laplace pyramid obtained by performing a 3-layer Laplace pyramid decomposition on the visible light original image; the three infrared detail layers include: a 0th infrared detail layer, a 1st infrared detail layer, and a 2nd infrared detail layer, which are respectively the 0th Laplace layer L0, the 1st Laplace layer L1, and the 2nd Laplace layer L2 in the Laplace pyramid obtained by performing a 3-layer Laplace pyramid decomposition on the infrared original image.

[0130] like Figure 1 As shown, in some embodiments of the present invention, the image processing method further includes: step S7, performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image; wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1.

[0131] In some embodiments of the present invention, N is greater than or equal to 1, and N is less than or equal to 5.

[0132] For details, please refer to Figure 4 When the visible light original image is decomposed into a three-layer Laplace pyramid, the obtained 0th Laplace layer L0, the 1st Laplace layer L1, and the 2nd Laplace layer L2 are the 0th visible light detail layer, the 1st visible light detail layer, and the 2nd visible light detail layer, respectively; when the infrared original image is decomposed into a three-layer Laplace pyramid, the obtained 0th Laplace layer L0, the 1st Laplace layer L1, and the 2nd Laplace layer L2 are the 0th infrared detail layer, the 1st infrared detail layer, and the 2nd infrared detail layer, respectively.

[0133] Specific as Figure 4 In some embodiments shown in the figure, the image processing method further includes: fusing the 0th visible light detail layer and the 0th infrared detail layer to obtain the 0th detail fused image F0 (eg Figure 4 The first visible light detail layer and the first infrared detail layer are fused to obtain the first detail fused image F1 (as shown in the image F0 in FIG. 1 ). Figure 4 The second visible light detail layer and the second infrared detail layer are fused to obtain a second detail fused image F2 (as shown in the image F1 in FIG. 1 ). Figure 4 (as shown in image F2).

[0134] like Figure 6 As shown, in some embodiments of the present invention, step S7, performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image includes: step S72: performing image fusion on the ath visible light detail layer and the ath infrared detail layer according to the ath detail fusion weight to obtain the ath detail fused image.

[0135] Specific as Figure 6 As shown, in some embodiments, step S7, performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image, further comprises: step S71, obtaining the ath detail fusion weight.

[0136] refer to Figure 7 , Figure 7 A schematic flow chart showing the steps of obtaining detail fusion weights in the image processing method of the present invention is shown.

[0137] In some embodiments of the present invention, in step S71, the step of obtaining the a-th detail fusion weight includes:

[0138] Step S711: obtaining the activity of the ath infrared detail layer according to the ath infrared detail layer;

[0139] Step S712: obtaining the activity of the ath visible light detail layer according to the ath visible light detail layer;

[0140] Step S713: Obtaining the detail fusion weight of the ath layer according to the activity of the ath infrared detail layer and the activity of the ath visible light detail layer, in combination with the ath infrared noise reduction parameter and the ath visible light noise reduction parameter.

[0141] For example, the filter kernel of the mean filter is a 1×5 filter kernel: In other embodiments, when resources are sufficient, a larger filter kernel may be selected.

[0142] In some embodiments, in the step of obtaining the activity of the detail layer, the absolute value of the detail layer is mean filtered to obtain the activity of the detail layer. Specifically, step S711: according to the a-th infrared detail layer, in the step of obtaining the activity of the a-th infrared detail layer, the absolute value of the a-th infrared detail layer is mean filtered to obtain the activity act_ir of the a-th infrared detail layer: act_ir=blur(abs(L2i); step S712: according to the a-th visible light detail layer, in the step of obtaining the activity of the a-th visible light detail layer, the absolute value of the a-th visible light detail layer is mean filtered to obtain the activity act_vi of the a-th visible light detail layer: act_vi=blur(abs(L2v)).

[0143] Step S713: In the step of obtaining the a-th layer detail fusion weight, the a-th infrared noise reduction parameter and the a-th visible light noise reduction parameter are suitable for adjusting the parameters of the weights of the a-th infrared detail layer and the a-th visible light detail layer respectively to suppress noise. In some embodiments, when the original image noise is large, adjusting the a-th infrared noise reduction parameter and the a-th visible light noise reduction parameter can help reduce noise.

[0144] like Figure 4 As shown, after obtaining the ath detail fusion weight, the ath visible light detail layer and the ath infrared detail layer are fused according to the ath detail fusion weight to obtain the ath detail fused image.

[0145] Specifically, after the step of obtaining the ath detail fusion weight, the ath detail layer fusion image Lar=Lav*weight is calculated according to the detail fusion weight, each infrared pixel value in the infrared detail layer, and each visible light pixel value in the visible light detail layer. Lavi weight+Lai*(1-weight Lavi weight), Lar represents the fused image of the a-th detail layer, weight Lavi represents the ath detail fusion weight, Lav is the ath visible light detail layer, and Lai is the ath infrared detail layer.

[0146] like Figure 4In some of the embodiments shown, the second visible light detail layer is the second Laplacian layer L2 in the Laplacian pyramid obtained by performing a three-layer Laplacian pyramid decomposition on the visible light original image; the second infrared detail layer is the second Laplacian layer L2 in the Laplacian pyramid obtained by performing a three-layer Laplacian pyramid decomposition on the infrared original image.

[0147] The second Laplacian layer L2 contains large edges in the image and has relatively less noise. The weights can be calculated based on the activity, and then fused according to the weights (Weighted Average, WA).

[0148] According to the ath infrared detail layer, the steps of obtaining the activity of the ath infrared detail layer are as follows: calculate the absolute value of the second infrared detail layer L2i to obtain the second infrared detail layer abs(L2i); use a 1x5 filter kernel to perform mean filtering on the second infrared detail layer abs(L2i) to obtain the activity act_ir of the second infrared detail layer L2i: act_ir=blur(abs(L2i)), blur is the filter function, and abs is the absolute value function.

[0149] According to the ath visible light detail layer, the steps of obtaining the activity of the ath visible light detail layer are as follows: calculate the absolute value of the second visible light detail layer L2v to obtain the second visible light detail layer abs(L2v); use a 1x5 filter kernel to mean filter the second visible light detail layer abs(L2v) to obtain the activity act_v of the second visible light detail layer L2v: act_v=blur(abs(L2v)), blur is the filter function, and abs is the absolute value function.

[0150] It should be noted that the filter kernel of the 1x5 mean filter is: Using a row-based filter core can save design time for line buffers. If resources are sufficient, a larger filter core can also be selected.

[0151] After that, calculate the second detail fusion weight:

[0152]

[0153] Among them, weight L2vi Expressed as the second detail fusion weight, k L2vi and k L2ir k represents the second visible light noise reduction parameter and the second infrared noise reduction parameter, respectively, and is an integer between 1 and 64, with an initial value of 1. When the noise of one of the images of the second visible light detail layer and the second infrared detail layer is large, the second visible light noise reduction parameter k is adjusted. L2vi and the second infrared noise reduction parameter k L2irThen, according to the detail fusion weight, the infrared pixel values ​​in the infrared detail layer, and the visible light pixel values ​​in the visible light detail layer, the second detail layer fusion image L2r = L2v*weight is calculated. L2vi +L2i*(1-weight L2vi ), L2r represents the fused image of the second detail layer, weight L2vi Represents the second detail fusion weight, L2v is the second visible light detail layer, and L2i is the second infrared detail layer.

[0154] After that, calculate the first detail fusion weight:

[0155]

[0156] Among them, weight L1vi Expressed as the first detail fusion weight, k L1vi and k L1ir k represents the first visible light noise reduction parameter and the first infrared noise reduction parameter, respectively, and is an integer between 1 and 64, with an initial value of 1. When the noise of one of the first visible light detail layer and the first infrared detail layer is large, the first visible light noise reduction parameter k is adjusted. L1vi and the first infrared noise reduction parameter k L1ir Then, according to the detail fusion weight, the infrared pixel values ​​in the infrared detail layer, and the visible light pixel values ​​in the visible light detail layer, the first detail layer fusion image L1r = L1v*weight is calculated. L1vi +L1i*(1-weight L1vi ), L1r represents the fused image of the first detail layer, weight L1vi It represents the first detail fusion weight, L1v is the first visible light detail layer, and L1i is the first infrared detail layer.

[0157] After that, calculate the 0th detail fusion weight:

[0158]

[0159] Among them, weight L0vi Denoted as the 0th detail fusion weight, k L0vi and k L0ir k represents the 0th visible light noise reduction parameter and the 0th infrared noise reduction parameter, respectively, and takes an integer between 1 and 64, with an initial value of 1. When the noise of one of the 0th visible light detail layer and the 0th infrared detail layer is large, adjust the 0th visible light noise reduction parameter k L0vi and the 0th infrared noise reduction parameter k L0irThen, according to the detail fusion weight, the infrared pixel values ​​in the infrared detail layer, and the visible light pixel values ​​in the visible light detail layer, the 0th detail layer fusion image L0r = L0v*weight is calculated. L0vi +L0i*(1-weight L0vi ), L0r represents the 0th detail layer fusion image, weight L0vi It is represented as the 0th detail fusion weight, L0v is the 0th visible light detail layer, and L0i is the 0th infrared detail layer.

[0160] In the above scheme, the visible light detail layer is obtained by performing N-layer pyramid decomposition on the visible light original image, and the infrared detail layer is obtained by performing N-layer pyramid decomposition on the infrared original image, and the visible light detail layer and the infrared detail layer are fused. Since the visible light detail layer and the infrared detail layer include noise information, by setting different weight parameters and obtaining the fusion weight of the infrared detail layer and the fusion weight of the visible light detail layer according to the weight parameters and the activity, the influence of the noise in the visible light detail layer and the infrared detail layer on the fusion effect can be effectively suppressed, and the fusion effect of the visible light detail layer and the infrared detail layer can be guaranteed.

[0161] It should be noted that, in some embodiments of the present invention, based on the ath detail fusion weight, the ath visible light detail layer and the ath infrared detail layer are fused to obtain the ath detail fused image. However, this approach is only an example.

[0162] In other embodiments of the present invention, in the step of performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image, the ath visible light detail layer and the ath infrared detail layer are averagely fused to obtain the ath detail fused image.

[0163] Continue to refer Figure 1 In some embodiments of the present invention, the image processing method further includes: step 8, performing image restoration based on the base layer fusion image to obtain a grayscale fusion image.

[0164] In some embodiments, step S1, performs N-layer pyramid decomposition on the visible light original image to obtain N layers of visible light detail layers; step S2, performs N-layer pyramid decomposition on the infrared original image to obtain N layers of infrared detail layers; the image processing method also includes: step 7, performs image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image; step 8, performs image restoration based on the base layer fused image to obtain a grayscale fused image, in which image restoration is performed based on the base layer fused image, the 0th detail fused image, ..., the ath detail fused image, ..., the N-1th detail fused image to obtain a grayscale fused image.

[0165] For example, Figure 4 In some of the embodiments shown, in the step of performing image restoration based on the base layer fusion image to obtain a grayscale fusion image, image restoration is performed based on the base layer fusion image, the second detail fusion image, the first detail fusion image and the 0th detail fusion image to obtain a grayscale fusion image.

[0166] In some embodiments, step S8, performing image restoration based on the base layer fusion image, and obtaining a grayscale fusion image comprises: obtaining the nth layer restoration image GRn, 0≤n≤N; wherein the step of obtaining the nth layer restoration image GRn comprises: upsampling the n+1th layer restoration image; and obtaining the nth layer restoration image GRn based on the sum of the upsampled n+1th layer restoration image and the nth detail fusion image.

[0167] In some embodiments, step S8, performing image restoration based on the base layer fusion image, and the step of obtaining a grayscale fusion image also includes: sequentially obtaining the nth layer restoration image GRn, the n+1th layer restoration image GRn+1, ..., until the 0th layer restoration image GR0, wherein the 0th layer restoration image GR0 is the grayscale fusion image.

[0168] It should be noted that, in step S8, in the step of performing image restoration based on the base layer fused image to obtain a grayscale fused image, the base layer fused image is used as the Nth restored image GRN.

[0169] Specific as Figure 4 and Figure 8 In some embodiments shown, in step S8, in the step of performing image restoration according to the base layer fused image to obtain a grayscale fused image, the third layer restored image GR3, the second layer restored image GR2, the first layer restored image GR1 and the 0th layer restored image GR0 obtained in sequence are respectively:

[0170]

[0171] The base layer fusion image F3 is used as the Nth restored image GR3; the third restored image GR3 is upsampled to obtain a sampled third restored image GR3↑, and the second restored image GR2 is obtained according to the sampled third restored image GR3↑ and L2; the second restored image GR2 is upsampled to obtain a sampled second restored image GR2↑, and the first restored image GR1 is obtained according to the sampled second restored image GR2↑ and L1; the first restored image GR1 is upsampled to obtain a sampled first restored image GR1↑, and the 0th restored image GR0 is obtained according to the sampled first restored image GR1↑ and L0.

[0172] Among them, the 0th restored image GR0 is a grayscale fused image.

[0173] refer to Fig. 9 , compare the effects of the technical solution of the present invention with other technologies, and discuss the technical effects of the present invention again, Fig. 9 A schematic diagram showing the comparison of the effects of fused images obtained by using different fusion methods in the image processing method of the present invention is shown, Fig. 9 The middle (a) image is the original visible light image. Fig. 9 The middle (b) image is the original infrared image. Fig. 9 The middle (c) image is a fused image obtained by using the existing technology (such as average fusion). Fig. 9 Figure (d) is a fused image obtained by using the fusion method of the present invention.

[0174] It can be seen that the image effect of this scheme is like "injecting" the infrared thermal target into the visible light image, so it can better preserve the details and contrast of the visible light image while highlighting the infrared thermal target.

[0175] like Fig.10 As shown, Fig.10 Figure (a) is a grayscale fused image obtained in some embodiments of the present invention, that is, according to the infrared pixel weight curve and the visible light pixel weight curve, the visible light base layer and the infrared base layer are fused to obtain a base layer fused image, thereby obtaining a grayscale fused image; Fig.10 The middle figure (b) is a grayscale fused image obtained by averaging the visible light base layer and the infrared base layer.

[0176] Compare Fig.10 (a) Figure and Fig.10 From Figure (b), we can see that Fig.10 The middle (a) figure highlights the infrared thermal targets, e.g. Fig.10 In (a), the kettle can be clearly identified from the background, and Fig.10 Other details in (a) can also be preserved, while Fig.10In the figure (b), the traditional average fusion method is adopted, which is prone to insufficient contrast of the target image, especially for the target image with opposite brightness in the visible light base layer and the infrared base layer, such as Fig.10 The black kettle in the middle (b) figure is placed in front of a white background. Due to its high temperature, it is brighter than the background in the infrared image. After using the traditional average fusion method, the fused image has low contrast and cannot highlight the hot target.

[0177] Therefore, the above scheme is adopted to obtain the visible light base layer by performing N-layer pyramid decomposition on the visible light original image; obtain the infrared base layer by performing N-layer pyramid decomposition on the infrared original image, and fuse the visible light base layer and the infrared base layer according to the infrared pixel weight curve and the visible light pixel weight curve, so that the display weights of different pixel values ​​of the infrared base layer in the base layer fused image are high and low, thereby achieving the effect of "injecting" the infrared base layer into the visible light base layer, which can effectively improve the prominence of the infrared target after fusion; and because the visible light base layer and the infrared base layer only include structural information, the prominent display of the infrared base layer in the base layer fused image can effectively avoid the introduction of noise information, and can effectively ensure the fusion effect of the visible light base layer and the infrared base layer.

[0178] It should be noted that some of the above embodiments are black and white fusion algorithms for the visible light original image and the infrared original image, that is, the obtained fused image is a grayscale fused image, that is, the grayscale fused image is black and white. In other embodiments of the present invention, colorization can also be performed based on the visible light original image and the infrared original image to obtain a color image.

[0179] refer to Fig.11 , shows a flow chart of another embodiment of the image processing method of the present invention.

[0180] The present invention will not repeat the same points as the above embodiments. The difference from the above embodiments is that in some embodiments of the present invention, colorization is performed based on the visible light original image and the infrared original image to obtain a color image.

[0181] Step S1, performing N-layer pyramid decomposition on the visible light original image to obtain a visible light base layer; then, step S2, performing N-layer pyramid decomposition on the infrared original image to obtain an infrared base layer; then, step S3, performing image fusion on the visible light base layer and the infrared base layer according to the infrared pixel weight curve and the visible light pixel weight curve to obtain a base layer fused image; then, step S8, performing image restoration according to the base layer fused image to obtain a grayscale fused image.

[0182] Step S1, performing N-layer pyramid decomposition on the visible light original image to obtain the visible light base layer, performing the step of performing Laplace pyramid decomposition on the visible light original image for the jth time to obtain the m-1th visible Gaussian layer; Step S2, performing N-layer pyramid decomposition on the infrared original image to obtain the infrared base layer, performing the step of performing Laplace pyramid decomposition on the infrared original image for the jth time to obtain the m-1th infrared Gaussian layer.

[0183] like Fig.11 As shown, in some embodiments of the present invention, the image processing method also includes: step S9, obtaining two chromaticity layers according to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer; step S10, upsampling the two chromaticity layers m-1 times respectively to obtain two chromaticity channel images; step S11, obtaining a color image according to the two chromaticity channel images and the grayscale fused image.

[0184] In some embodiments, in step S9, in the step of obtaining two chromaticity layers according to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer, two chromaticity layers are obtained according to the N-1th visible Gaussian layer and the N-1th infrared Gaussian layer. The N-1th visible Gaussian layer and the N-1th infrared Gaussian layer are Gaussian layers obtained during the last execution of the image decomposition step. During the last execution of the image decomposition step, the noise in the obtained Gaussian layer is less, so that the noise of the obtained chromaticity layer is smaller and storage resources are saved.

[0185] In some specific embodiments, the image processing method also includes: setting pseudo color parameters; step S9, in the step of obtaining two chromaticity layers according to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer, two chromaticity layers are obtained according to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer in combination with the pseudo color parameters.

[0186] For example, in the step of setting the pseudo color parameters, the pseudo color parameters may be set according to the color style.

[0187] Specifically, the step of setting the pseudo-color parameters includes: setting the pseudo-color parameters of one chromaticity layer corresponding to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer respectively; setting the pseudo-color parameters of another chromaticity layer corresponding to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer respectively.

[0188] In the step of obtaining two chromaticity layers according to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer in combination with the pseudo color parameters, two chromaticity layers are obtained according to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer in combination with the corresponding pseudo color parameters. Specifically:

[0189] U_Lm-1 =G m-1 e_vi*q1-G m-1 e_ir*q2;

[0190] V_L m-1 =G m-1 e_vi*q3-G m-1 e_ir*q4.

[0191] Among them, U_L m-1 is the U channel chrominance layer corresponding to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer, V_L m-1 is the V channel chromaticity layer corresponding to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer, G m-1 e_vi is the m-1th visible Gaussian layer, G m-1 e_ir is the m-1th infrared Gaussian layer, q1 is the pseudo-color parameter corresponding to the U channel chromaticity layer and the m-1th visible Gaussian layer, q2 is the pseudo-color parameter corresponding to the U channel chromaticity layer and the m-1th infrared Gaussian layer, q3 is the pseudo-color parameter corresponding to the V channel chromaticity layer and the m-1th visible Gaussian layer, and q4 is the pseudo-color parameter corresponding to the V channel chromaticity layer and the m-1th infrared Gaussian layer. In some exemplary embodiments, in step S1, in the step of performing N-layer pyramid decomposition on the visible light original image, a 3-layer pyramid decomposition is performed on the visible light original image; in step S1, in the step of performing N-layer pyramid decomposition on the infrared original image, a 3-layer pyramid decomposition is performed on the infrared original image; in the step of obtaining two chromaticity layers, the obtained chromaticity layers are respectively:

[0192] U_L2 = G2e_vi*q1-G2e_ir*q2;

[0193] V_L2=G2e_vi*q3-G2e_ir*q4.

[0194] Among them, U_L2 is the U channel chromaticity layer corresponding to the second visible Gaussian layer and the second infrared Gaussian layer, V_L2 is the V channel chromaticity layer corresponding to the second visible Gaussian layer and the second infrared Gaussian layer, G2e_vi is the second visible Gaussian layer, G2e_ir is the second infrared Gaussian layer, q1 is the pseudo-color parameter corresponding to the U channel chromaticity layer and the second visible Gaussian layer, q2 is the pseudo-color parameter corresponding to the U channel chromaticity layer and the second infrared Gaussian layer, q3 is the pseudo-color parameter corresponding to the V channel chromaticity layer and the second visible Gaussian layer, and q4 is the pseudo-color parameter corresponding to the V channel chromaticity layer and the second infrared Gaussian layer.

[0195] In some embodiments, in step S10, upsampling is performed m-1 times on the two chroma layers respectively to obtain two chroma channel images.

[0196] Specifically, for the U channel chromaticity layer U_L corresponding to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer m-1 Upsample and get U_L m-2 =sample_up(U_L m-1 ), where U_L m-2 It is the U channel chrominance layer corresponding to the m-2th visible Gaussian layer and the m-2th infrared Gaussian layer, and sample_up is the upsampling process.

[0197] Specifically, for the V channel chromaticity layer V_L corresponding to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer m-1 Upsample to get V_L m-2 =sample_up(V_L m-1 ), where V_L m-2 It is the V channel chroma layer corresponding to the m-2th visible Gaussian layer and the m-2th infrared Gaussian layer, and sample_up is the upsampling process.

[0198] The above sampling results are used as two chromaticity channel images, and the two chromaticity channel images are respectively: U chromaticity channel image U Lm-2out and V chroma channel image V Lm-2ouz , that is U Lm-2out =U_L m-2 , V L0out =V_L m-2 .

[0199] In some other embodiments, after obtaining the U channel chromaticity layer U_L corresponding to the m-2th visible Gaussian layer and the m-2th infrared Gaussian layer, m-2 Then, the U channel chromaticity layer U_L corresponding to the m-2 visible Gaussian layer and the m-2 infrared Gaussian layer is processed. m-2 Upsampling is performed until the U channel chromaticity layer U_L0 corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer is obtained, U_L0 = sample_up(U_L1), wherein the 0th visible Gaussian layer and the 0th infrared Gaussian layer are the Gaussian layers obtained during the first execution of the image decomposition step, and U_L1 is the U channel chromaticity layer corresponding to the 1st visible Gaussian layer and the 1st infrared Gaussian layer.

[0200] In some other embodiments, after obtaining the V channel chromaticity layer V_L corresponding to the m-2th visible Gaussian layer and the m-2th infrared Gaussian layer, m-2 Then, the V channel chromaticity layer V_L corresponding to the m-2 visible Gaussian layer and the m-2 infrared Gaussian layer is processed. m-2Upsampling is performed until the V channel chromaticity layer V_L0 corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer is obtained, V_L0 = sample_up(V_L1), wherein the 0th visible Gaussian layer and the 0th infrared Gaussian layer are the Gaussian layers obtained during the first execution of the image decomposition step, and V_L1 is the V channel chromaticity layer corresponding to the 1st visible Gaussian layer and the 1st infrared Gaussian layer.

[0201] The above sampling results are used as two chromaticity channel images, and the two chromaticity channel images are respectively: U chromaticity channel image U L0out and V chroma channel image V L0out , that is U L0out =U_L0,V L0out =V_L0.

[0202] In some exemplary embodiments, in step S1, in the step of performing N-layer pyramid decomposition on the visible light original image, a 3-layer pyramid decomposition is performed on the visible light original image; in step S1, in the step of performing N-layer pyramid decomposition on the infrared original image, a 3-layer pyramid decomposition is performed on the infrared original image; in step S10, in the step of upsampling the two chromaticity layers m-1 times respectively to obtain two chromaticity channel images, the two chromaticity layers are upsampled 2 times respectively to obtain two chromaticity channel images.

[0203] Upsample the U channel chromaticity layer U_L2 corresponding to the second visible Gaussian layer and the second infrared Gaussian layer to obtain the U channel chromaticity channel image U_L1 corresponding to the first visible Gaussian layer and the first infrared Gaussian layer; upsample the U channel chromaticity channel image U_L1 corresponding to the first visible Gaussian layer and the first infrared Gaussian layer to obtain the U channel chromaticity channel image U_L0 corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer; upsample the V channel chromaticity channel image V_L2 corresponding to the second visible Gaussian layer and the second infrared Gaussian layer to obtain the V channel chromaticity channel image V_L1 corresponding to the first visible Gaussian layer and the first infrared Gaussian layer; upsample the V channel chromaticity channel image V_L1 corresponding to the first visible Gaussian layer and the first infrared Gaussian layer to obtain the V channel chromaticity channel image V_L0 corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer. The specific upsampling process is as follows:

[0204] U_L1 = sample_up(U_L2);

[0205] U_L0 = sample_up(U_L1);

[0206] V_L1 = sample_up(V_L2);

[0207] V_L0=sample_up(V_L1).

[0208] Among them, U_L2 is the U channel chromaticity layer corresponding to the 2nd visible Gaussian layer and the 2nd infrared Gaussian layer, U_L1 is the U channel chromaticity channel image corresponding to the 1st visible Gaussian layer and the 1st infrared Gaussian layer, U_L0 is the U channel chromaticity channel image corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer, V_L1 is the V channel chromaticity channel image corresponding to the 1st visible Gaussian layer and the 1st infrared Gaussian layer, V_L0 is the V channel chromaticity channel image corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer, and sample_up is the upsampling process.

[0209] The above sampling results are used as two chromaticity channel images, and the two chromaticity channel images are respectively: U chromaticity channel image U L0out and V chroma channel image V L0out , that is U L0out =U_L0,V L0out =V_L0.

[0210] In some embodiments, step S11, a color image is obtained according to the two chromaticity channel images and the grayscale fused image.

[0211] Specifically, the image processing method also includes: using the grayscale fused image as the brightness channel image; using the above sampling results as two chromaticity channel images; step S11, obtaining a color image based on the two chromaticity channel images and the grayscale fused image, and obtaining a color image based on the two chromaticity channel images and the brightness channel image.

[0212] like Figure 4 In some embodiments shown in the figure, in step S1, in the step of performing N-layer pyramid decomposition on the visible light original image, the visible light original image is decomposed into 3 layers of pyramids; in step S1, in the step of performing N-layer pyramid decomposition on the infrared original image, the infrared original image is decomposed into 3 layers of pyramids; in step S11, in the step of obtaining a color image according to the two chromaticity channel images and the grayscale fused image, according to U chromaticity channel image U L0out and V chroma channel image V L0out and the brightness channel image Y L0out , get the color image in YUV space.

[0213] Specifically, in step S8, after restoring the base layer fusion image, a 0th layer restoration image GR0 is obtained, wherein the 0th layer restoration image GR0 is the grayscale fusion image, that is, Y L0out =GR0.

[0214] Combination Figure 4 refer to Fig.12 , compared to Figure 4 Grayscale fusion image in, Fig.12Middle (a) Figure, Fig.12 (b) Figure and Fig.12 The images in (c) are all color images, which expands the application scenarios of image fusion and is more Fig.12 Middle (a) Figure, Fig.12 (b) Figure and Fig.12 As can be seen from Figure (c), by adjusting the pseudo-color parameters, color images with different color styles are obtained, making the obtained color images have a natural feel.

[0215] refer to Fig.13 , shows a flow chart of another embodiment of the image processing method of the present invention.

[0216] The present invention will not repeat the same points as the above-mentioned embodiments. The difference from the above-mentioned embodiments is that in some embodiments of the present invention, the image processing method can also perform a pinch-off operation on the obtained detail layer and limit the size of the pixel value of the obtained detail layer to alleviate the halo effect.

[0217] The halo effect is an optical defect on an image; it usually appears at the edges of an image where there is a sharp change in light and dark.

[0218] In some embodiments of the present invention, an N-layer pyramid decomposition is performed on a visible light original image to obtain N layers of visible light detail layers; an N-layer pyramid decomposition is performed on an infrared original image to obtain N layers of infrared detail layers; wherein N is greater than or equal to 1; the image processing method further comprises: step S7, in image fusion of the ath visible light detail layer and the ath infrared detail layer, a pinch-off operation is performed on the ath visible light detail layer and the ath infrared detail layer respectively; step S7, in the step of obtaining the ath detail fused image, image fusion is performed on the ath visible light detail layer and the ath infrared detail layer that have been pinched off, to obtain the ath detail fused image.

[0219] like Fig.13 As shown, in some embodiments of the present invention, the image processing method includes: step S1: performing N-layer pyramid decomposition on the visible light original image, and also obtaining N layers of visible light detail layers; then, step S110: performing a pinch-off operation on the a-th visible light detail layer; step S2: performing N-layer pyramid decomposition on the infrared original image, and also obtaining N layers of infrared detail layers; then, step S210: performing a pinch-off operation on the a-th infrared detail layer; then, step S220: performing image fusion on the a-th visible light detail layer that has been pinched off and the a-th infrared detail layer that has been pinched off, to obtain the a-th detail fused image.

[0220] Specifically, the pinch-off operation is to set an upper threshold and a lower threshold of the pixel value of the visible light detail layer, and to set an upper threshold and a lower threshold of the pixel value of the infrared detail layer. In some embodiments, the pixel value range of the visible light detail layer is [-256, 255]; the pixel value range of the infrared detail layer is [-256, 255].

[0221] In some embodiments of the examples, continue to refer to Figure 4 In step S1, in the step of performing N-layer pyramid decomposition on the visible light original image, the visible light original image is decomposed into 3 layers of pyramids; in step S1, in the step of performing N-layer pyramid decomposition on the infrared original image, the infrared original image is decomposed into 3 layers of pyramids; the 0th Laplace layer L0, the 1st Laplace layer L1, and the 2nd Laplace layer L2 are obtained. Figure 4 The Laplacian layers in the Laplacian pyramid on the left side correspond to the 0th visible light detail layer, the 1st visible light detail layer, and the 2nd visible light detail layer. Figure 4 The Laplace layers in the Laplace pyramid on the middle right correspond to the 0th visible light detail layer, the 1st visible light detail layer, and the 2nd visible light detail layer; set the upper and lower thresholds of the pixel values ​​of the 0th visible light detail layer; set the upper and lower thresholds of the pixel values ​​of the 1st visible light detail layer; set the upper and lower thresholds of the pixel values ​​of the 2nd visible light detail layer; set the upper and lower thresholds of the pixel values ​​of the 0th infrared detail layer; set the upper and lower thresholds of the pixel values ​​of the 1st infrared detail layer; set the upper and lower thresholds of the pixel values ​​of the 2nd infrared detail layer.

[0222] For example, Fig.14 ,in Fig.14 Figure (a) shows a grayscale fused image obtained without using the pinch-off operation in some embodiments of the present invention. Fig.14 Figure (b) is a grayscale fused image obtained by using a pinch-off operation in some embodiments of the present invention.

[0223] Compare Fig.14 (a) Figure and Fig.14 As can be seen from Figure (b), the grayscale fused image obtained by pinching off the visible light detail layer and the infrared detail layer and then performing image fusion can suppress the influence of the halo effect and reduce resource usage.

[0224] refer to Fig.15 , showing a schematic diagram of another embodiment of the image processing method of the present invention.

[0225] The present invention will not repeat the same points as the above embodiments. The difference from the above embodiments is that in some embodiments of the present invention, the number of endpoints for determining the infrared pixel weight curve may be greater than 4.

[0226] like Fig.15 As shown, Fig.15 The infrared pixel weight curve is obtained by connecting the first endpoint A1 and the second endpoint A2, and the second endpoint A2 and the third endpoint A3 to obtain the infrared pixel weight curve in the cold target area.

[0227] The third endpoint A3 and the fourth endpoint A4 are connected to obtain an infrared pixel weight curve of the background target area.

[0228] The fourth endpoint A4 and the fifth endpoint A5, and the fifth endpoint A5 and the sixth endpoint A6 are connected to obtain an infrared pixel weight curve in the thermal target area.

[0229] refer to Fig.16 ,in Fig.16 The middle (a) figure is the grayscale fusion image obtained by the infrared pixel weight curve determined by the four endpoints. Fig.16 The middle figure (b) is a grayscale fusion image obtained based on the infrared pixel weight curve determined by the six endpoints.

[0230] Compare Fig.16 Figure (a) and Fig.16 In the middle (b) figure, the method of determining the infrared pixel weight curve according to the six endpoints not only improves the fusion ratio of the infrared base layer, but also makes the infrared pixel weight curve adjustment more refined. The infrared thermal target in the obtained grayscale fusion image is more prominent, for example Fig.16 In the middle (b) image, the kettle with higher temperature stands out more in the fused image.

[0231] Correspondingly, the present invention also provides an image processing device, comprising: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the processor executes the steps of the above-mentioned image processing method.

[0232] In the image processing device, the processor executes the computer program stored in the memory to perform the steps of the image processing method of the present invention. The specific technical solution of the image processing device can refer to the embodiment of the aforementioned image processing method, and the present invention will not be repeated here.

[0233] Correspondingly, the present invention also provides a computer-readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, on which computer instructions are stored, and when the computer instructions are executed, the steps of the image processing method described above are executed.

[0234] The computer-readable storage medium stores computer instructions for executing the steps of the image processing method of the present invention when running. The specific technical solution of the computer-readable storage medium refers to the embodiment of the aforementioned image processing method, and the present invention will not be repeated here.

[0235] In summary, the visible light base layer is obtained by performing N-layer pyramid decomposition on the visible light original image; the infrared base layer is obtained by performing N-layer pyramid decomposition on the infrared original image, and the visible light base layer and the infrared base layer are fused according to the infrared pixel weight curve and the visible light pixel weight curve, so that the display weights of different pixel values ​​of the infrared base layer in the base layer fused image are high and low, thereby achieving the effect of "injecting" the infrared base layer into the visible light base layer, which can effectively improve the prominence of the infrared target after fusion; and because the visible light base layer and the infrared base layer only include structural information, the prominent display of the infrared base layer in the base layer fused image can effectively avoid the introduction of noise information, and can effectively ensure the fusion effect of the visible light base layer and the infrared base layer.

[0236] Moreover, by setting the background target area, the hot target area and the cold target area within the pixel value range of the infrared base layer, and the infrared pixel weights corresponding to the infrared pixel values ​​in different target areas are different, the display weights of the infrared pixel values ​​in different target areas in the base layer fused image are high and low, thereby ensuring that the infrared base layer can be highlighted in the base layer fused image; and the infrared pixel weights corresponding to the infrared pixel values ​​in the cold target area and the infrared pixel weights corresponding to the infrared pixel values ​​in the hot target area are both greater than the infrared pixel weights corresponding to the infrared pixel values ​​in the background target area, which can effectively improve the contrast and naturalness of the fused image and effectively improve the fusion effect.

[0237] In addition, the present invention sets an initial weight value corresponding to the minimum infrared pixel value, an initial infrared pixel weight corresponding to the first infrared pixel value, an initial infrared pixel weight corresponding to the second infrared pixel value, and an initial infrared pixel weight corresponding to the maximum infrared pixel value, that is, sets different initial infrared pixel weights, and obtains infrared pixel weight curves in different target areas according to different initial infrared pixel weights. Subsequently, different infrared weight curves can be obtained by adjusting the infrared pixel weights corresponding to different infrared pixel values, thereby optimizing the infrared weight curve, so that the infrared base layer can be accurately highlighted in the base layer fused image.

[0238] In addition, the technical solution of the present invention performs N-layer image decomposition on the visible light original image and the infrared original image to obtain a visible light detail layer and an infrared detail layer, and performs image fusion on the ath visible light detail layer and the ath infrared detail layer according to the ath detail fusion weight. Since the visible light detail layer and the infrared detail layer include noise information, and the calculation of the detail fusion weight can adjust the noise in the visible light detail layer and the infrared detail layer, using the detail fusion weight for image fusion can reduce the noise in the visible light detail layer and the infrared detail layer, thereby ensuring the fusion effect of the visible light detail layer and the infrared detail layer.

[0239] In addition, the technical solution of the present invention performs a pinch-off operation on the infrared detail layer and the visible light detail layer, that is, performs threshold limiting processing on the infrared detail layer and the visible light detail layer, so that the pixel values ​​in the infrared detail layer and the visible light detail layer are limited, thereby reducing resource usage and suppressing the generation of halo effect.

[0240] In addition, the technical solution of the present invention obtains two chromaticity channel images according to the visible Gaussian layer and the infrared Gaussian layer obtained after decomposing the N-layer pyramid, and performs color component processing according to the two chromaticity channel images and the grayscale fusion image to obtain a color output image, thereby expanding the application scenarios of image fusion.

[0241] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. An image processing method, It is characterized in that include: Perform N-layer pyramid decomposition on the original visible light image to obtain the visible light base layer; Perform N-layer pyramid decomposition on the original infrared image to obtain an infrared base layer, where N is a natural number; According to the infrared pixel weight curve and the visible light pixel weight curve, the visible light base layer and the infrared base layer are image fused to obtain a base layer fused image.

2. The image processing method according to claim 1, It is characterized in that The image processing method further comprises: setting an infrared pixel weight curve; The steps to set up an infrared pixel weight curve include: A background target area, a hot target area and a cold target area are set within the infrared base layer pixel value range, wherein the infrared pixel value in the background target area is between the infrared pixel value in the hot target area and the infrared pixel value in the cold target area; An infrared pixel weight curve for the background target field, an infrared pixel weight curve for the hot target field, and an infrared pixel weight curve for the cold target field are respectively set, wherein the infrared pixel weight corresponding to the infrared pixel value in the background target field is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the hot target field; the infrared pixel weight corresponding to the infrared pixel value in the background target field is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the cold target field.

3. The image processing method according to claim 2, It is characterized in that In the step of setting a background target area, a hot target area and a cold target area within the infrared base layer pixel value range, an average value of the infrared base layer pixel values ​​is located within the background target area.

4. The image processing method according to claim 2, It is characterized in that An initial infrared pixel weight corresponding to a minimum infrared pixel value, an initial infrared pixel weight corresponding to a first infrared pixel value, an initial infrared pixel weight corresponding to a second infrared pixel value, and an initial infrared pixel weight corresponding to a maximum infrared pixel value are respectively set, wherein the minimum infrared pixel value, the first infrared pixel value, the second infrared pixel value, and the maximum infrared pixel value increase in sequence, the initial infrared pixel weight corresponding to the minimum infrared pixel value is greater than or equal to the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the maximum infrared pixel value is greater than or equal to the initial infrared pixel weight corresponding to the second infrared pixel value, and the area between the minimum infrared pixel value and the first infrared pixel value is the cold target area, the area between the first infrared pixel value and the second infrared pixel value is the background target area, and the area between the second infrared pixel value and the maximum infrared pixel value is the hot target area; According to the initial infrared pixel weight corresponding to the minimum infrared pixel value and the initial infrared pixel weight corresponding to the first infrared pixel value, an infrared pixel weight curve of the cold target field is obtained; according to the initial infrared pixel weight corresponding to the first infrared pixel value and the initial infrared pixel weight corresponding to the second infrared pixel value, an infrared pixel weight curve of the background target field is obtained; An infrared pixel weight curve in the thermal target area is obtained according to the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value.

5. The image processing method according to claim 2, It is characterized in that The steps of obtaining the base layer fusion image include: According to the infrared pixel weight curve, the visible light pixel weight curve, the visible light pixel values ​​in the visible light base layer and the infrared pixel values ​​in the infrared base layer, the base layer fused image Gnr=w_vis*Gnv+w_ir*Gni is calculated, where Gnr represents the base layer fused image, w_vis represents the visible light pixel weight, w_ir represents the infrared pixel weight, Gnv represents the visible light pixel value, Gni represents the infrared pixel value, and n∈N, where n is a natural number.

6. The image processing method according to claim 1, It is characterized in that Perform N-layer pyramid decomposition on the original visible light image to obtain N layers of visible light detail layers; Perform N-layer pyramid decomposition on the original infrared image to obtain N-layer infrared detail layers; Wherein, N is greater than or equal to 1; The image processing method further comprises: The ath visible light detail layer and the ath infrared detail layer are image-fused to obtain the ath detail fused image; wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1.

7. The image processing method according to claim 6, It is characterized in that In the step of fusing the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image, the ath visible light detail layer and the ath infrared detail layer are averagely fused to obtain the ath detail fused image.

8. The image processing method according to claim 6, It is characterized in that The step of fusing the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image includes: fusing the ath visible light detail layer and the ath infrared detail layer according to the ath detail fusion weight to obtain the ath detail fused image.

9. The image processing method according to claim 8, It is characterized in that The steps of obtaining the a-th detail fused image include: The detail layer fusion image Lar = Lav*weight is calculated based on the ath detail fusion weight, the infrared pixel values ​​in the infrared detail layer, and the visible light pixel values ​​in the visible light detail layer. Lavi +Lai*(1-weight Lavi ), Lar represents the fused image of the ath detail layer, weight Lavi represents the ath detail fusion weight, Lav is the ath visible light detail layer, and Lai is the ath infrared detail layer.

10. The image processing method according to claim 8, It is characterized in that The step of fusing the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image further includes: obtaining the ath detail fusion weight.

11. The image processing method according to claim 10, It is characterized in that The steps of obtaining the a-th detail fusion weight include: According to the a-th infrared detail layer, the activity of the a-th infrared detail layer is obtained; According to the a-th visible light detail layer, obtaining the activity of the a-th visible light detail layer; According to the activity of the ath infrared detail layer and the activity of the ath visible light detail layer, in combination with the ath infrared noise reduction parameter and the ath visible light noise reduction parameter, the ath layer detail fusion weight is obtained.

12. The image processing method according to claim 6, It is characterized in that Also includes: Before the step of fusing the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image: performing pinch-off operations on the ath visible light detail layer and the ath infrared detail layer respectively; In the step of obtaining the ath detail fused image, the ath visible light detail layer subjected to the pinch-off operation and the ath infrared detail layer subjected to the pinch-off operation are fused to obtain the ath detail fused image.

13. The image processing method according to claim 1, It is characterized in that The image processing method further includes: performing image restoration according to the base layer fused image to obtain a grayscale fused image.

14. The image processing method according to claim 13, It is characterized in that Perform N-layer pyramid decomposition on the visible light original image, and obtain N layers of visible light detail layers; Perform N-layer pyramid decomposition on the original infrared image to obtain N-layer infrared detail layers; Wherein, N is greater than or equal to 1; The image processing method further includes: fusing the ath visible light detail layer and the ath infrared detail layer to obtain the ath detail fused image; wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1; In the step of obtaining a grayscale fused image, image restoration is performed based on the base layer fused image, the 0th detail fused image, ..., the ath detail fused image, ..., the N-1th detail fused image to obtain a grayscale fused image.

15. The image processing method according to claim 1, It is characterized in that In at least one of the step of performing N-layer pyramid decomposition on the visible light original image and the step of performing N-layer pyramid decomposition on the infrared original image, performing N-layer Laplacian pyramid decomposition on the image to be processed; The image to be processed is one of the visible light original image and the infrared original image.

16. The image processing method according to claim 15, It is characterized in that The step of performing N-layer Laplace pyramid decomposition includes: performing the Laplace pyramid decomposition step at least once, performing the Laplace pyramid decomposition step for the jth time, and obtaining the j-1th Laplace layer, where j is a positive integer greater than or equal to 0 and less than or equal to N; The m-th execution of the Laplace pyramid decomposition step includes: Perform Gaussian filtering on the m-1th layer image; Down-sampling the Gaussian-filtered m-1th layer image by 1 / 2 to obtain the sampled and filtered mth layer image; Upsample the m-th layer image to obtain the m-1-th Gaussian layer; According to the m-1th Gaussian layer and the m-1th layer image, an m-1th Laplace layer is obtained, where m is a positive integer greater than or equal to 2 and less than or equal to N.

17. The image processing method according to claim 16, It is characterized in that In the step of performing the Laplacian pyramid decomposition for the first time, Gaussian filtering is performed on the image to be processed to obtain the 0th Laplacian layer.

18. The image processing method according to claim 16, It is characterized in that The image processing method further comprises: performing image restoration according to the base layer fused image to obtain a grayscale fused image; According to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer, two chromaticity layers are obtained; The two chroma layers are respectively upsampled m-1 times to obtain two chroma channel images; A color image is obtained according to the two chromaticity channel images and the grayscale fused image.

19. The image processing method according to claim 18, It is characterized in that In the step of obtaining two chromaticity layers according to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer, two chromaticity layers are obtained according to the N-1th visible Gaussian layer and the N-1th infrared Gaussian layer.

20. An image processing device, It is characterized in that include: A memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the processor executes the steps of the image processing method according to any one of claims 1 to 19.

21. A computer-readable storage medium, It is characterized in that The computer-readable storage medium is a non-volatile storage medium or a non-transitory storage medium, on which computer instructions are stored. When the computer instructions are executed, the steps of the image processing method described in any one of claims 1 to 19 are executed.

Images