Image processing method and image processing apparatus therefor, and computer-readable storage medium

Abstract

An image processing method and an image processing apparatus therefor, and a computer-readable storage medium. The image processing method comprises: performing N-layer pyramid decomposition on a visible light original image to obtain a visible light base layer; performing N-layer pyramid decomposition on an infrared original image to obtain an infrared base layer; on the basis of an infrared pixel weighting curve and a visible light pixel weighting curve, performing image fusion on the visible light base layer and the infrared base layer to obtain a base layer fused image. The present application performs N-layer pyramid decomposition on a visible light original image to obtain a visible light base layer, performs N-layer pyramid decomposition on an infrared original image to obtain an infrared base layer, and fuses the visible light base layer and the infrared base layer on the basis of an infrared pixel weighting curve and a visible light pixel weighting curve, such that display weightings of different pixel values of the infrared base layer in the base layer fused image vary, thus effectively improving the degree of prominence of infrared targets after fusion.

Description

Image processing method, image processing device thereof, and computer readable storage medium

[0001] The present application claims priority to the Chinese patent application No. 202411170243X, filed on August 23, 2024, and titled "Image processing method, image processing device thereof, and computer readable storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of image acquisition, in particular to an image processing method, an image processing device thereof, and a computer readable storage medium. BACKGROUND

[0003] Infrared imaging is not affected by the intensity of ambient light and can work in a weak light environment, so infrared images are widely used in target recognition and detection fields, but the texture details are relatively blurred; while visible light images can provide rich texture details. After fusing the visible light image and the infrared image, the obtained fused image can have the advantages of both.

[0004] However, the fusion effect of the prior art visible light image and infrared image is not good, for example, the infrared target is not prominent enough, the image lacks naturalness, and the noise is large.

[0005] SUMMARY

[0006] The technical problem solved by the present application is how to improve the fusion effect of the visible light image and the infrared image.

[0007] To solve the above technical problem, an image processing method is provided, comprising: performing N-layer pyramid decomposition on a visible light original image to obtain a visible light base layer; performing N-layer pyramid decomposition on an infrared original image to obtain an infrared base layer, wherein N is a natural number; and performing image fusion on the visible light base layer and the infrared base layer according to an infrared pixel weight curve and a visible light pixel weight curve to obtain a base layer fused image.

[0008] Optionally, the image processing method further comprises: setting an infrared pixel weight curve; the step of setting the infrared pixel weight curve comprises: setting a background target region, a hot target region and a cold target region in the infrared base layer pixel value range, wherein the infrared pixel value in the background target region is between the infrared pixel value in the hot target region and the infrared pixel value in the cold target region; setting the infrared pixel weight curve of the background target region, the infrared pixel weight curve of the hot target region and the infrared pixel weight curve of the cold target region respectively, wherein the infrared pixel weight corresponding to the infrared pixel value in the background target region is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the hot target region; the infrared pixel weight corresponding to the infrared pixel value in the background target region is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the cold target region.

[0009] Optionally, in the step of setting the background target region, the hot target region and the cold target region in the infrared base layer pixel value range, the average value of the infrared base layer pixel value is in the background target region.

[0010] Optionally, the initial infrared pixel weight corresponding to the minimum infrared pixel value, the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value are set respectively, wherein the minimum infrared pixel value, the first infrared pixel value, the second infrared pixel value and the maximum infrared pixel value increase in turn, the initial infrared pixel weight corresponding to the minimum infrared pixel value is greater than or equal to the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the maximum infrared pixel value is greater than or equal to the initial infrared pixel weight corresponding to the second infrared pixel value, and the minimum infrared pixel value and the first infrared pixel value are in the cold target region, the first infrared pixel value and the second infrared pixel value are in the background target region, and the second infrared pixel value and the maximum infrared pixel value are in the hot target region; the infrared pixel weight curve of the cold target region is obtained according to the initial infrared pixel weight corresponding to the minimum infrared pixel value and the initial infrared pixel weight corresponding to the first infrared pixel value; the infrared pixel weight curve of the background target region is obtained according to the initial infrared pixel weight corresponding to the first infrared pixel value and the initial infrared pixel weight corresponding to the second infrared pixel value; and the infrared pixel weight curve of the hot target region is obtained according to the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value.

[0011] Optionally, the step of obtaining the base layer fusion image comprises: calculating the base layer fusion image Gnr = w_vis*Gnv + w_ir*Gni according to the infrared pixel weight curve, the visible light pixel weight curve, each visible light pixel value in the visible light base layer and each infrared pixel value in the infrared base layer, wherein Gnr represents the base layer fusion image, w_vis represents the visible light pixel weight, w_ir represents the infrared pixel weight, Gnv represents the visible light pixel value, Gni represents the infrared pixel value, and n is a natural number.

[0012] Optionally, the visible light original image is subjected to N-layer pyramid decomposition, and N visible light detail layers are obtained; the infrared original image is subjected to N-layer pyramid decomposition, and N infrared detail layers are obtained; wherein N is greater than or equal to 1; the image processing method further comprises: performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain an ath detail fusion image; wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1.

[0013] Optionally, in the step of performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain an ath detail fusion image, the ath visible light detail layer and the ath infrared detail layer are subjected to average fusion to obtain the ath detail fusion image.

[0014] Optionally, the step of performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain an ath detail fusion image comprises: performing image fusion on the ath visible light detail layer and the ath infrared detail layer according to the ath detail fusion weight to obtain the ath detail fusion image.

[0015] Optionally, the step of obtaining the ath detail fusion image comprises: calculating the detail layer fusion image Lar = Lav*weight Lavi + Lai*(1-weight Lavi ) according to the ath detail fusion weight, each infrared pixel value in the infrared detail layer and each visible light pixel value in the visible light detail layer, wherein Lar represents the ath detail layer fusion image, weight Lavi represents the ath detail fusion weight, Lav represents the ath visible light detail layer, and Lai represents the ath infrared detail layer.

[0016] Optionally, the step of performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain an ath detail fusion image further comprises: obtaining the ath detail fusion weight.

[0017] Optionally, the step of obtaining the a-th detail fusion weight comprises: obtaining an activity of the a-th infrared detail layer according to the a-th infrared detail layer; obtaining an activity of the a-th visible light detail layer according to the a-th visible light detail layer; and obtaining the a-th detail fusion weight according to the activities of the a-th infrared detail layer and the a-th visible light detail layer, the a-th infrared denoising parameter and the a-th visible light denoising parameter.

[0018] Optionally, the image processing method further comprises: before the step of obtaining the a-th detail fusion image by fusing the a-th visible light detail layer and the a-th infrared detail layer, performing a clipping operation on the a-th visible light detail layer and the a-th infrared detail layer respectively; and in the step of obtaining the a-th detail fusion image, fusing the a-th visible light detail layer and the a-th infrared detail layer after the clipping operation to obtain the a-th detail fusion image.

[0019] Optionally, the image processing method further comprises: performing image restoration according to the base layer fusion image to obtain a gray-scale fusion image.

[0020] Optionally, the visible light original image is decomposed into N layers of visible light detail layers by N-layer pyramid decomposition, and the infrared original image is decomposed into N layers of infrared detail layers by N-layer pyramid decomposition, wherein N is greater than or equal to 1; the image processing method further comprises: fusing the a-th visible light detail layer and the a-th infrared detail layer to obtain the a-th detail fusion image, wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1; and in the step of obtaining the gray-scale fusion image, performing image restoration according to the base layer fusion image, the 0-th detail fusion image, the a-th detail fusion image, and the N-1-th detail fusion image to obtain the gray-scale fusion image.

[0021] Optionally, in at least one of the steps of decomposing the visible light original image into N layers of visible light detail layers by N-layer pyramid decomposition and decomposing the infrared original image into N layers of infrared detail layers by N-layer pyramid decomposition, N-layer Laplacian pyramid decomposition is performed on a to-be-processed image, wherein the to-be-processed image is one of the visible light original image and the infrared original image.

[0022] Optionally, the step of performing N-layer Laplacian pyramid decomposition comprises: performing the Laplacian pyramid decomposition step at least once, and the j-th time of performing the Laplacian pyramid decomposition step obtains a j-1-th Laplacian layer, j being a positive integer greater than or equal to 0 and less than or equal to N; wherein the m-th time of performing the Laplacian pyramid decomposition step comprises: performing Gaussian filtering on an m-1-th layer image; performing 1 / 2 down-sampling on the m-1-th layer image after the Gaussian filtering to obtain an m-1-th Gaussian layer; and obtaining an m-1-th Laplacian layer according to the m-1-th Gaussian layer and the m-1-th layer image, m being a positive integer greater than or equal to 2 and less than or equal to N.

[0023] Optionally, in the step of performing Laplacian pyramid decomposition for the first time, Gaussian filtering is performed on the image to be processed to obtain a 0th Laplacian layer.

[0024] Optionally, the image processing method further comprises: performing image restoration on the basis layer fusion image to obtain a gray-scale fusion image; obtaining two chroma layers according to the (m-1)th visible Gaussian layer and the (m-1)th infrared Gaussian layer; performing (m-1) times of up-sampling on the two chroma layers respectively to obtain two chroma channel images; and obtaining a color image according to the two chroma channel images and the gray-scale fusion image.

[0025] Optionally, in the step of obtaining two chroma layers according to the (m-1)th visible Gaussian layer and the (m-1)th infrared Gaussian layer, two chroma layers are obtained according to the (N-1)th visible Gaussian layer and the (N-1)th infrared Gaussian layer.

[0026] Correspondingly, the present application also provides an image processing device, comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and when the processor executes the program, the processor executes the steps of the image processing method described above.

[0027] Correspondingly, the present application also provides a computer readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, and stores computer instructions, wherein the computer instructions run to execute the steps of the image processing method described above.

[0028] Compared with the prior art, the technical scheme of the embodiment of the present application has the following beneficial effects:

[0029] In the technical scheme of the present application, the visible light original image is subjected to N-layer pyramid decomposition to obtain a visible light basis layer, the infrared original image is subjected to N-layer pyramid decomposition to obtain an infrared basis layer, and the visible light basis layer and the infrared basis layer are fused according to an infrared pixel weight curve and a visible light pixel weight curve, so that the display weight of different pixel values of the infrared basis layer in the basis layer fusion image is high or low, thereby achieving the effect of “injecting” the infrared basis layer into the visible light basis layer, which can effectively improve the prominence of the infrared target after fusion.

[0030] Further, by setting the background target field, the hot target field and the cold target field in the infrared base layer pixel value range, and the infrared pixel weight corresponding to the infrared pixel value in different target fields being different, the display weight of the infrared pixel value in different target fields in the base layer fusion image exists high and low, thereby ensuring that the infrared base layer can be highlighted in the base layer fusion image; and the infrared pixel weight corresponding to the infrared pixel value in the cold target field and the infrared pixel weight corresponding to the infrared pixel value in the hot target field are both greater than the infrared pixel weight corresponding to the infrared pixel value in the background target field, which can effectively improve the contrast and naturalness of the fused image, and effectively improve the fusion effect.

[0031] Further, the present application sets the initial weight value corresponding to the minimum infrared pixel value, the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value, that is, sets different initial infrared pixel weights, and obtains the infrared pixel weight curve in different target fields according to different initial infrared pixel weights, and then the infrared pixel weight corresponding to different infrared pixel values can be adjusted to obtain different infrared weight curves, so as to realize the optimization of the infrared weight curve and make the infrared base layer accurately highlighted in the base layer fusion image.

[0032] Further, the present application obtains the visible light detail layer and the infrared detail layer by decomposing the visible light original image and the infrared original image into N layers, and performs image fusion on the a-th visible light detail layer and the a-th infrared detail layer according to the a-th detail fusion weight. Since the visible light detail layer and the infrared detail layer include noise information, and the calculation of the detail fusion weight can adjust the noise in the visible light detail layer and the infrared detail layer, the image fusion using the detail fusion weight can reduce the noise in the visible light detail layer and the infrared detail layer, thereby ensuring the fusion effect of the visible light detail layer and the infrared detail layer.

[0033] Further, the present application limits the threshold of the infrared detail layer and the visible light detail layer, that is, performs clipping operation on the infrared detail layer and the visible light detail layer, so that the pixel values in the infrared detail layer and the visible light detail layer are limited, resource occupation is reduced, and the generation of the halo effect is inhibited.

[0034] Further, the present application obtains two chroma channel images according to the visible Gaussian layer and the infrared Gaussian layer obtained after N-layer pyramid decomposition, and performs color component processing on the two chroma channel images and the gray-scale fusion image to obtain a color output image, thereby expanding the application scenarios of image fusion. BRIEF DESCRIPTION OF DRAWINGS

[0035] Fig. 1 is a flowchart of an embodiment of the image processing method of the present application;

[0036] Fig. 2 is a flowchart of the step of N-layer pyramid decomposition in an embodiment of the image processing method of the present application;

[0037] Fig. 3 is a schematic diagram of the algorithm of N-layer pyramid decomposition in an embodiment of the image processing method of the present application;

[0038] Fig. 4 is a simulation diagram of N-layer pyramid decomposition in an embodiment of the image processing method of the present application;

[0039] Fig. 5 is a schematic diagram of an infrared pixel weight curve in an embodiment of the image processing method of the present application;

[0040] Fig. 6 is a flowchart of the step of detail layer fusion in an embodiment of the image processing method of the present application;

[0041] Fig. 7 is a flowchart of the step of obtaining detail fusion weight in an embodiment of the image processing method of the present application;

[0042] Fig. 8 is a schematic diagram of the image restoration process in an embodiment of the image processing method of the present application;

[0043] Fig. 9 is a comparison diagram of the effects of the fusion images obtained by using different fusion methods in an embodiment of the image processing method of the present application;

[0044] Fig. 10 is a comparison diagram of the effects of the fusion images obtained by using different fusion methods for the base layer in an embodiment of the image processing method of the present application;

[0045] Fig. 11 is a flowchart of the step of obtaining colorization processing in an embodiment of the image processing method of the present application;

[0046] Fig. 12 is a simulation diagram of the colorization processing in an embodiment of the image processing method of the present application;

[0047] Fig. 13 is a flowchart of the pinch operation in an embodiment of the image processing method of the present application;

[0048] Fig. 14 is a simulation diagram of the pinch operation processing in an embodiment of the image processing method of the present application;

[0049] Fig. 15 is a schematic diagram of another infrared pixel weight curve in an embodiment of the image processing method of the present application;

[0050] Fig. 16 is a simulation diagram of image fusion in an embodiment of the image processing method of the present application. DETAILED DESCRIPTION

[0051] The current image fusion algorithm has the following shortcomings: 1. When the image fusion is carried out, the noise of the image is also fused, resulting in that the fused image has large noise; 2. The infrared thermal target after fusion is not prominent enough; 3. The contrast of the fused image is not as good as the original visible light image, and the natural sense of the fused image is not as good as the original visible light image; 4. The color of the fused image lacks natural sense after colorization.

[0052] To solve the above technical problems, the present application provides an image processing method, which comprises the following steps:

[0053] According to the infrared pixel weight curve and the visible light pixel weight curve, the visible light base layer and the infrared base layer are fused, so that the display weight of the infrared base layer in the base layer fused image is high or low, thereby ensuring the prominence degree of the infrared base layer and improving the fusion effect of the visible light base layer and the infrared base layer; and since the visible light base layer and the infrared base layer only include structural information, noise information is avoided to be introduced, thereby improving the fusion effect of the visible light base layer and the infrared base layer.

[0054] To make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0055] Referring to FIG. 1, a flowchart of an embodiment of the image processing method of the present application is shown.

[0056] The image processing method comprises:

[0057] Step S1: performing N-layer pyramid decomposition on the visible light original image to obtain a visible light base layer;

[0058] Step S2: performing N-layer pyramid decomposition on the infrared original image to obtain an infrared base layer, wherein N is a natural number;

[0059] Step S3: according to the infrared pixel weight curve and the visible light pixel weight curve, fusing the visible light base layer and the infrared base layer to obtain a base layer fused image.

[0060] It should be noted that the visible light original image and the infrared original image are obtained by two different detectors. Specifically, the two images are basically registered by mechanical means, and then accurately registered by image scaling, stretching, cropping and the like.

[0061] Firstly, at least one of step S1 and step S2 is performed to carry out N-layer pyramid decomposition to obtain a base layer. Specifically, step S1 and step S2 can be performed in sequence; or step S2 and step S1 can be performed in sequence; or step S1 and step S2 can be performed simultaneously.

[0062] In some embodiments of the present application, in the step of carrying out N-layer pyramid decomposition, at least one of the step of carrying out N-layer Laplacian pyramid decomposition on the visible light original image (step S1) and the step of carrying out N-layer pyramid decomposition on the infrared original image (step S2) is the step of carrying out N-layer Laplacian pyramid decomposition on the to-be-processed image, wherein the to-be-processed image is one of the visible light original image and the infrared original image.

[0063] Specifically, in the step of carrying out N-layer pyramid decomposition on the visible light original image (step S1), N-layer Laplacian pyramid decomposition is carried out on the visible light original image; and in the step of carrying out N-layer pyramid decomposition on the infrared original image (step S2), N-layer Laplacian pyramid decomposition is carried out on the infrared original image.

[0064] In some embodiments of the example, the step of carrying out N-layer Laplacian pyramid decomposition comprises: performing the step of carrying out Laplacian pyramid decomposition at least once, performing the step of carrying out Laplacian pyramid decomposition for the jth time to obtain a (j-1)th Laplacian layer, j being a positive integer greater than or equal to 0 and less than or equal to N-1.

[0065] The technical solution of carrying out N-layer Laplacian pyramid decomposition in an embodiment of the image processing method of the present application will be described in detail below with reference to FIG. 2, FIG. 3 and FIG. 4.

[0066] Referring to FIG. 2, a flowchart of performing the step of carrying out Laplacian pyramid decomposition for the mth time in an embodiment of the image processing method of the present application is shown.

[0067] Specifically, performing the step of carrying out Laplacian pyramid decomposition for the mth time comprises: step S11, performing Gaussian filtering on an (m-1)th layer image; step S12, performing 1 / 2 downsampling on the (m-1)th layer image subjected to Gaussian filtering to obtain a sampled and filtered mth layer image; step S13, performing upsampling on the mth layer image to obtain an (m-1)th Gaussian layer; and step S14, obtaining an (m-1)th Laplacian layer according to the (m-1)th Gaussian layer and the (m-1)th layer image, wherein m is a positive integer greater than or equal to 2 and less than or equal to N-1.

[0068] In some embodiments, in the step of performing Gaussian filtering on the (m-1)th layer image (step S11), the filter kernel of the Gaussian filtering is a 3x3 filter kernel:

[0069] Specifically, the downsampling and the Gaussian filtering before the downsampling are performed in the following manner: first, edge padding and Gaussian filtering are performed, and then 1 / 2 downsampling is performed, wherein the edge padding is performed by copying the edges to fill one row and one column at each of the four edges of the image; the Gaussian filtering is performed by using the filter kernel as above, and then the values of the even rows and even columns are reserved as the downsampling result (counting from the 0th row and 0th column, the 0th row is regarded as an even row, and the 0th column is regarded as an even column) when the 1 / 2 downsampling is performed.

[0070] Specifically, the upsampling is performed in the following manner: one row of 0 is inserted after each row of the image, and one row of 0 is inserted after each column of the image, to obtain an image with double width and height, and it is noted that the odd rows and odd columns are 0 (counting from the 0th row and 0th column, the 0th row is regarded as an even row, and the 0th column is regarded as an even column).

[0071] In other embodiments of the present application, when resources are sufficient, the filter kernel of the Gaussian filtering in the step of performing Gaussian filtering on the (m-1)th layer image can also be a 5*5 filter kernel.

[0072] In the above scheme, the fusion image obtained by using the 5*5 filter kernel has a more natural edge transition.

[0073] In combination with reference to FIG. 3 and FIG. 4, wherein FIG. 3 shows an algorithm schematic diagram of performing N-layer pyramid decomposition in the image processing method of the present application; and FIG. 4 shows a simulation schematic diagram of performing N-layer pyramid decomposition in the image processing method of the present application.

[0074] As shown in FIG. 3, in the first time of performing the Laplacian pyramid decomposition step, Gaussian filtering is performed on the 0th layer image G0 The 0th layer image after Gaussian filtering is subjected to 1 / 2 downsampling to obtain a sampled and filtered 1st layer image G1; the 1st layer image after sampling and filtering is subjected to upsampling to obtain a 0th Gaussian layer G0e; and the 0th Gaussian layer G0e and the 0th layer image G0 are used to obtain a 0th Laplacian layer L0.

[0075] That is, L0=G0-G0e.

[0076] It is to be noted that in some embodiments, the step of performing the Laplacian pyramid decomposition for the first time includes performing Gaussian filtering on a to-be-processed image to obtain a 0th Laplacian layer, wherein the to-be-processed image is one of the visible light original image and the infrared original image.

[0077] Specifically, in the process of N-layer pyramid decomposition of the visible light original image, the first time of performing the Laplacian pyramid decomposition step includes: performing Gaussian filtering on the visible light original image (as shown by the image G0 on the left side in FIG. 4) performing 1 / 2 down-sampling on the Gaussian-filtered 0th layer image to obtain a sampled and filtered 1st layer image G1 (as shown by the image G1 on the left side in FIG. 4); performing up-sampling on the sampled and filtered 1st layer image G1 to obtain a 0th Gaussian layer G0e (as shown by the image G0e on the left side in FIG. 4); and obtaining a 0th Laplacian layer L0 (as shown by the image L0 on the left side in FIG. 4) according to the 0th Gaussian layer G0e and the 0th layer image G0.

[0078] Specifically, in the process of N-layer pyramid decomposition of the visible light original image, the first time of performing the Laplacian pyramid decomposition step includes: performing Gaussian filtering on the visible light original image (as shown by the image G0 on the left side in FIG. 4) performing 1 / 2 down-sampling on the Gaussian-filtered 0th layer image to obtain a sampled and filtered 1st layer image G1 (as shown by the image G1 on the left side in FIG. 4); performing up-sampling on the sampled and filtered 1st layer image G1 to obtain a 0th Gaussian layer G0e (as shown by the image G0e on the left side in FIG. 4); and obtaining a 0th Laplacian layer L0 (as shown by the image L0 on the left side in FIG. 4) according to the 0th Gaussian layer G0e and the 0th layer image G0.

[0079] With reference back to FIG. 3, in the second time of performing the Laplacian pyramid decomposition step,

[0080] performing Gaussian filtering on the 1st layer image G1 performing 1 / 2 down-sampling on the Gaussian-filtered 1st layer image to obtain a sampled and filtered 2nd layer image G2; performing up-sampling on the sampled and filtered 2nd layer image G2 to obtain a 1st Gaussian layer G1e; and obtaining a 1st Laplacian layer L1 according to the 1st Gaussian layer G1e and the 1st layer image G1.

[0081] That is, L1=G1-G1e.

[0082] Specifically, in the process of N-layer pyramid decomposition of the visible light original image, the second time of performing the Laplacian pyramid decomposition step includes: performing Gaussian filtering on the 1st layer image G1 (as shown by the image G1 on the left side in FIG. 4) obtained based on the visible light original image down-sampling the 1st layer image G1 filtered by the Gaussian filter by 1 / 2 to obtain a 2nd layer image G2 filtered by the sampling (as shown in the image G2 on the left side of Fig. 4); up-sampling the 2nd layer image G2 filtered by the sampling to obtain a 1st Gaussian layer G1e (as shown in the image G1e on the left side of Fig. 4); and obtaining a 1st Laplacian layer L1 according to the 1st Gaussian layer G1e and the 1st layer image G1 (as shown in the image L1 on the left side of Fig. 4).

[0083] Specifically, in the process of performing the N-layer pyramid decomposition on the infrared original image, the 2nd time of performing the Laplacian pyramid decomposition step includes: performing a Gaussian filter on the 1st layer image G1 (as shown in the image G1 on the right side of Fig. 4) obtained based on the infrared original image down-sampling the 1st layer image G1 filtered by the Gaussian filter by 1 / 2 to obtain a 2nd layer image G2 filtered by the sampling (as shown in the image G2 on the right side of Fig. 4); up-sampling the 2nd layer image G2 filtered by the sampling to obtain a 1st Gaussian layer G1e (as shown in the image G1e on the right side of Fig. 4); and obtaining a 1st Laplacian layer L1 according to the 1st Gaussian layer G1e and the 1st layer image G1 (as shown in the image L1 on the right side of Fig. 4).

[0084] With reference to Fig. 3, in the 3rd time of performing the Laplacian pyramid decomposition step, a Gaussian filter is performed on the 2nd layer image G2 down-sampling the 2nd layer image G2 filtered by the Gaussian filter by 1 / 2 to obtain a 3rd layer image G3 filtered by the sampling; up-sampling the 3rd layer image G3 filtered by the sampling to obtain a 2nd Gaussian layer G2e; and obtaining a 2nd Laplacian layer L2 according to the 2nd Gaussian layer G2e and the 2nd layer image G2.

[0085] That is, L1=G2-G2e.

[0086] Specifically, in the process of performing the N-layer pyramid decomposition on the visible light original image, the 3rd time of performing the Laplacian pyramid decomposition step includes: performing a Gaussian filter on the 2nd layer image G2 (as shown in the image G2 on the left side of Fig. 4) obtained based on the visible light original image down-sampling the 2nd layer image G2 filtered by the Gaussian filter by 1 / 2 to obtain a 3rd layer image G3 filtered by the sampling (as shown in the image G3 on the left side of Fig. 4); up-sampling the 3rd layer image G3 filtered by the sampling to obtain a 2nd Gaussian layer G2e (as shown in the image G1e on the left side of Fig. 4); and obtaining a 2nd Laplacian layer L2 according to the 2nd Gaussian layer G2e and the 2nd layer image G2 (as shown in the image L2 on the left side of Fig. 4).

[0087] Specifically, in the step S2, the third time of performing the Laplacian pyramid decomposition step in the process of performing the N-layer pyramid decomposition on the infrared original image comprises: performing a Gaussian filtering on the second layer image G2 (as shown in the image G2 on the right side of the Fig. 4) obtained based on the infrared original image performing a 1 / 2 down-sampling on the second layer image G2 after the Gaussian filtering to obtain a sampled and filtered third layer image G3 (as shown in the image G3 on the right side of the Fig. 4); performing an up-sampling on the sampled and filtered third layer image G3 to obtain a second Gaussian layer G2e (as shown in the image G1e on the right side of the Fig. 4); and obtaining a second Laplacian layer L2 (as shown in the image L2 on the right side of the Fig. 4) according to the second Gaussian layer G2e and the second layer image G2.

[0088] The mth time of performing the Laplacian pyramid decomposition step is as described in the above solution, and thus the present application is not described here again by analogy.

[0089] In some embodiments of the example, in the step of performing the N-layer pyramid decomposition, a three-layer Laplacian pyramid decomposition is performed; and the step of performing the three-layer Laplacian pyramid decomposition comprises: performing the Laplacian pyramid decomposition step for three times.

[0090] After performing the three times of the Laplacian pyramid decomposition step, four layers of Gaussian pyramid and three layers of Laplacian pyramid are obtained, wherein the four layers of Gaussian pyramid comprise: the 0th layer image G0, the 1st layer image G1, the 2nd layer image G2 and the 3rd layer image G3, and the three layers of Laplacian pyramid comprise: the 0th Laplacian layer L0, the 1st Laplacian layer L1 and the 2nd Laplacian layer L2.

[0091] As shown in the Fig. 4, the 0th layer image G0 on the left side, the 1st layer image G1 on the left side, the 2nd layer image G2 on the left side and the 3rd layer image G3 on the left side are respectively the visible light original image and the corresponding sampled and filtered 1st layer image G1, the sampled and filtered 2nd layer image G2 and the sampled and filtered 3rd layer image G3 in the four layers of Gaussian pyramid obtained by performing the three-layer Laplacian pyramid decomposition on the visible light original image; the 0th Laplacian layer L0 on the left side, the 1st Laplacian layer L1 on the left side and the 2nd Laplacian layer L2 on the left side are respectively the 0th Laplacian layer L0, the 1st Laplacian layer L1 and the 2nd Laplacian layer L2 in the three layers of Laplacian pyramid obtained by performing the three-layer Laplacian pyramid decomposition on the visible light original image.

[0092] As shown in FIG. 4, the right 0th layer image G0, the right 1st layer image G1, the right 2nd layer image G2 and the right 3rd layer image G3 are respectively the infrared original image and the corresponding sampled filtered 1st layer image G1, the sampled filtered 2nd layer image G2 and the sampled filtered 3rd layer image G3 in the 4-layer Gaussian pyramid obtained by performing 3-layer Laplacian pyramid decomposition on the infrared original image; the left 0th layer image L0, the left 1st layer image L1 and the left 2nd layer image L2 are respectively the 0th Laplacian layer L0, the 1st Laplacian layer L1 and the 2nd Laplacian layer L2 in the 3-layer Laplacian pyramid obtained by performing 3-layer Laplacian pyramid decomposition on the infrared original image.

[0093] In some embodiments of the present application, at least one of the step S1 and the step S2 is performed, and in the step of performing N-layer pyramid decomposition, a detail layer is also obtained.

[0094] The base layer is a thumbnail of the original image, and after N-layer pyramid decomposition, the base layer contains very low noise information, almost no noise, and the base layer only carries the structural basic information of the original image.

[0095] The detail layer only carries texture, edge, noise and other information, and therefore the fusion strategy of the base layer is the most critical, which relates to the basic framework, contrast and highlight degree of the infrared target of the fusion image.

[0096] Specifically, with reference back to FIG. 1, in the step of performing N-layer pyramid decomposition, the base layer is obtained, that is, in the step S1, the N-layer pyramid decomposition is performed on the visible light original image to obtain a visible light base layer; and in the step S2, the N-layer pyramid decomposition is performed on the infrared original image to obtain an infrared base layer.

[0097] Specifically, with reference back to FIG. 1, in the step S1 of performing N-layer pyramid decomposition on the visible light original image, N-layer visible light detail layers are also obtained; and in the step S2 of performing N-layer pyramid decomposition on the infrared original image, N-layer infrared detail layers are also obtained.

[0098] In some embodiments of the present application, the highest layer of the Gaussian pyramid, that is, the sampled filtered Nth layer image, is the base layer; and each Laplacian layer in the Laplacian pyramid is a detail layer.

[0099] Specifically, the highest layer of the Gaussian pyramid obtained by performing N-layer Laplacian pyramid decomposition on the visible light original image is the visible light base layer; and the highest layer of the Gaussian pyramid obtained by performing N-layer Laplacian pyramid decomposition on the infrared original image is the infrared base layer.

[0100] Specifically, the visible light original image is decomposed into N layers of Laplacian pyramid to obtain N layers of visible light detail layers; and the infrared original image is decomposed into N layers of Laplacian pyramid to obtain N layers of infrared detail layers.

[0101] Specifically, the 0th, 1st and 2nd layers of the Laplacian pyramid obtained by decomposing the visible light original image into N layers of Laplacian pyramid are respectively the 0th, 1st and 2nd visible light detail layers; and the 0th, 1st and 2nd layers of the Laplacian pyramid obtained by decomposing the infrared original image into N layers of Laplacian pyramid are respectively the 0th, 1st and 2nd infrared detail layers.

[0102] With reference back to FIG. 1, after obtaining the visible light base layer and the infrared light base layer, the image processing method further comprises: step S3, performing image fusion on the visible light base layer and the infrared base layer according to the infrared pixel weight curve and the visible light pixel weight curve to obtain a base layer fusion image.

[0103] The infrared pixel weight curve is suitable for representing the proportion of the infrared pixel value in the obtained base layer fusion image. Based on the fusion of the visible light base layer and the infrared base layer according to the infrared pixel weight curve and the visible light pixel weight curve, the display weight of different pixel values of the infrared base layer in the base layer fusion image is high or low, so that the infrared base layer is “injected” into the visible light base layer, and the prominence of the infrared target after fusion can be effectively improved; and since the visible light base layer and the infrared base layer only include structural information, the prominent display of the infrared base layer in the base layer fusion image can effectively avoid the introduction of noise information, and can effectively ensure the fusion effect of the visible light base layer and the infrared base layer.

[0104] In some embodiments of the present application, as shown in FIG. 1, the image processing method further comprises: setting an infrared pixel weight curve.

[0105] Specifically, with reference back to FIG. 1, the step of setting the infrared pixel weight curve comprises:

[0106] Step S4: setting a background target region, a hot target region and a cold target region in the infrared base layer pixel value range, wherein the infrared pixel value in the background target region is between the infrared pixel value in the hot target region and the infrared pixel value in the cold target region;

[0107] Step S5: respectively setting the infrared pixel weight curve of the background target region, the infrared pixel weight curve of the hot target region and the infrared pixel weight curve of the cold target region.

[0108] wherein the infrared pixel weight corresponding to the infrared pixel value in the background target region is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the hot target region; and the infrared pixel weight corresponding to the infrared pixel value in the background target region is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the cold target region.

[0109] In some embodiments of the present application, in the step S4 of setting the background target region, the hot target region and the cold target region within the range of the infrared base layer pixel values, the average value Xmean of the infrared base layer pixel values is located within the background target region.

[0110] The average value Xmean of the pixel values is close to the background brightness, and the average value Xmean of the infrared base layer pixel values is close to the background brightness of the infrared base layer. In the background target region, the hot target region and the cold target region divided by locating the average value Xmean of the infrared base layer pixel values within the background target region, the background target region is located around the background brightness of the infrared base layer, and the hot target region and the cold target region are both deviated from the background brightness. Furthermore, the method of dividing the background target region, the hot target region and the cold target region by locating the average value Xmean of the infrared base layer pixel values within the background target region makes the set infrared pixel weight curve a dynamic curve, and the set infrared pixel weight curve will change with the change of the average value Xmean of the infrared base layer pixel values.

[0111] In some embodiments of the present application, the range of the infrared base layer pixel values is 0 to 256, the average value of the infrared base layer pixel values is real-time statistics, and when the absolute value of the difference between the current infrared base layer pixel value and the average value of the infrared base layer pixel values is greater than a preset difference threshold value, the current infrared base layer pixel value is located within the cold target region or the hot target region. For example, the preset difference threshold value is 40.

[0112] When the difference between the current infrared base layer pixel value and the average value of the infrared base layer pixel values is greater than a preset difference threshold value, and the difference is a positive number, the current infrared base layer pixel value is located within the hot target region.

[0113] When the difference between the current infrared base layer pixel value and the average value of the infrared base layer pixel values is greater than a preset difference threshold value, and the difference is a negative number, the current infrared base layer pixel value is located within the cold target region.

[0114] Referring to FIG. 5, FIG. 5 shows a schematic diagram of an infrared pixel weight curve in the image processing method of the present application. In the figure, the X-axis coordinate is represented as the infrared pixel value IR, and the Y-axis coordinate is represented as the infrared pixel weight weight.

[0115] In some embodiments of the present application, the step S5 of setting the infrared pixel weight curve of the background target region, the infrared pixel weight curve of the hot target region and the infrared pixel weight curve of the cold target region respectively comprises: setting the initial infrared pixel weight W1 corresponding to the minimum infrared pixel value Xmin, the initial infrared pixel weight W2 corresponding to the first infrared pixel value X1, the initial infrared pixel weight W3 corresponding to the second infrared pixel value X2 and the initial infrared pixel weight W4 corresponding to the maximum infrared pixel value Xmax respectively.

[0116] The minimum infrared pixel value Xmin, the first infrared pixel value X1, the second infrared pixel value X2 and the maximum infrared pixel value Xmax increase sequentially, the initial infrared pixel weight W1 corresponding to the minimum infrared pixel value Xmin is greater than or equal to the initial infrared pixel weight W2 corresponding to the first infrared pixel value X1, the initial infrared pixel weight W4 corresponding to the maximum infrared pixel value Xmax is greater than or equal to the initial infrared pixel weight W3 corresponding to the second infrared pixel value X2, and the minimum infrared pixel value Xmin and the first infrared pixel value X1 are in the cold target region, the first infrared pixel value X1 and the second infrared pixel value X2 are in the background target region, and the second infrared pixel value X2 and the maximum infrared pixel value Xmax are in the hot target region.

[0117] In some embodiments of the present application, the step S5 of setting the infrared pixel weight curve of the background target region, the infrared pixel weight curve of the hot target region and the infrared pixel weight curve of the cold target region respectively further comprises: obtaining the infrared pixel weight curve of the cold target region according to the initial infrared pixel weight corresponding to the minimum infrared pixel value and the initial infrared pixel weight corresponding to the first infrared pixel value; obtaining the infrared pixel weight curve of the background target region according to the initial infrared pixel weight corresponding to the first infrared pixel value and the initial infrared pixel weight corresponding to the second infrared pixel value; and obtaining the infrared pixel weight curve of the hot target region according to the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value.

[0118] As shown in FIG. 5, the infrared pixel weight curve of the hot target region and the infrared pixel weight curve of the cold target region are both greater than the infrared pixel weight curve of the background target region, that is, the weight of the infrared base layer of the target significantly deviating from the background brightness in the base layer fusion image is increased to appropriately highlight: the target whose pixel value is significantly greater than the average value Xmean of the infrared base layer pixel value, that is, the "hot target" whose temperature is significantly higher than the background temperature, is a more important target in the infrared image (for example, a car driving in a forest), which needs to be highlighted, and a higher weight is set; the target whose pixel value is significantly less than the average value Xmean of the infrared base layer pixel value, that is, the "cold target" whose temperature is significantly lower than the background temperature, is also a more important target in the infrared image (for example, a vehicle that has been stationary in a forest for a period of time), which also needs to be highlighted, and a higher weight is also set.

[0119] Specifically, as shown in FIG. 5, the abscissa of the first endpoint A1 is the minimum infrared pixel value Xmin, and the ordinate of the first endpoint A1 is the initial infrared pixel weight W1 corresponding to the minimum infrared pixel value Xmin; the abscissa of the second endpoint A2 is the first infrared pixel value X1, and the ordinate of the second endpoint A2 is the initial infrared pixel weight W2 corresponding to the first infrared pixel value X1; the abscissa of the third endpoint A3 is the second infrared pixel value X2, and the ordinate of the third endpoint A3 is the initial infrared pixel weight W3 corresponding to the second infrared pixel value X2; the abscissa of the fourth endpoint A4 is the maximum infrared pixel value Xmax, and the ordinate of the fourth endpoint A4 is the initial infrared pixel weight W4 corresponding to the maximum infrared pixel value Xmax.

[0120] With reference to FIG. 5, the acquisition of the infrared pixel weight curve is specifically connecting the first endpoint A1 and the second endpoint A2 to obtain the infrared pixel weight curve L1 of the cold target region; connecting the second endpoint A2 and the third endpoint A3 to obtain the infrared pixel weight curve L2 of the background target region; and connecting the third endpoint A3 and the fourth endpoint A4 to obtain the infrared pixel weight curve L3 of the hot target region.

[0121] In some embodiments as shown in FIG. 5, the coordinates of the first endpoint A1 are (0.3, 0), the coordinates of the second endpoint A2 are (0.1, xmean-dx1), the coordinates of the third endpoint A3 are (0, xmean+dx2), and the coordinates of the fourth endpoint A4 are (0.9, 255), where dx1 is the distance between xmean and the abscissa of the second endpoint A2, and dx2 is the distance between xmean and the abscissa of the third endpoint A3.

[0122] With reference back to FIG. 4, after the infrared pixel weight curve is obtained, each infrared pixel in the infrared base layer is "injected" into the visible light base layer according to the infrared pixel weight corresponding to each infrared pixel in the infrared pixel weight curve, which can effectively improve the prominence of the infrared target after fusion and effectively improve the contrast and naturalness of the image after fusion, where the image after fusion of the infrared base layer and the visible light base layer is F3.

[0123] In the above scheme, by setting the initial weight value corresponding to the minimum infrared pixel value, the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the second infrared pixel value, and the initial infrared pixel weight corresponding to the maximum infrared pixel value, i.e., setting different initial infrared pixel weights, and obtaining the infrared pixel weight curve in different target fields according to the different initial infrared pixel weights, different infrared weight curves can be obtained by adjusting the infrared pixel weight corresponding to different infrared pixel values, which realizes the optimization of the infrared weight curve and can effectively improve the prominence of the infrared target after fusion.

[0124] With reference back to FIG. 1, in some embodiments of the present application, the image processing method further comprises: step S6: setting a visible light pixel weight curve.

[0125] In some embodiments of the present application, the sum of the visible light pixel weight and the infrared pixel weight is 1 for the same pixel value; in the step of setting the visible light pixel weight curve in step S6, the visible light pixel weight curve is set according to the infrared pixel weight curve.

[0126] In some embodiments of the example, the weight of each visible light pixel in the visible light pixel weight curve is 1-w_ir, where w_ir is each infrared pixel weight.

[0127] With reference back to FIG. 1, in some embodiments, in the step of performing image fusion on the visible light base layer and the infrared base layer to obtain a base layer fusion image in step S3, the base layer fusion image Gnr is calculated according to the infrared pixel weight curve, the visible light pixel weight curve, each visible light pixel value in the visible light base layer, and each infrared pixel value in the infrared base layer: Gnr=w_vis*Gnv+w_ir*Gni, where Gnr represents the base layer fusion image, w_vis represents the visible light pixel weight, w_ir represents the infrared pixel weight, Gnv represents the visible light pixel value, Gni represents the infrared pixel value, and n∈N, n is a natural number.

[0128] As shown in FIG. 1, in some embodiments of the present application, the image processing method further comprises: step S1, performing N-layer pyramid decomposition on the visible light original image to obtain N-layer visible light detail layers, wherein the N-layer visible light detail layers comprise: a 0th visible light detail layer, a 1st visible light detail layer, …, an (N-1)th visible light detail layer; step S2, performing N-layer pyramid decomposition on the infrared original image to obtain N-layer infrared detail layers, wherein the N-layer infrared detail layers comprise: a 0th infrared detail layer, a 1st infrared detail layer, …, an (N-1)th infrared detail layer.

[0129] As shown in some embodiments of FIG. 4, N=3; the 3-layer visible light detail layers comprise: a 0th visible light detail layer, a 1st visible light detail layer and a 2nd visible light detail layer, which are respectively a 0th Laplacian layer L0, a 1st Laplacian layer L1 and a 2nd Laplacian layer L2 in a Laplacian pyramid obtained by performing 3-layer Laplacian pyramid decomposition on the visible light original image; the 3-layer infrared detail layers comprise: a 0th infrared detail layer, a 1st infrared detail layer and a 2nd infrared detail layer, which are respectively a 0th Laplacian layer L0, a 1st Laplacian layer L1 and a 2nd Laplacian layer L2 in a Laplacian pyramid obtained by performing 3-layer Laplacian pyramid decomposition on the infrared original image.

[0130] As shown in FIG. 1, in some embodiments of the present application, the image processing method further comprises: step S7, performing image fusion on the a-th visible light detail layer and the a-th infrared detail layer to obtain an a-th detail fusion image; wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1.

[0131] In some embodiments of the present application, N is greater than or equal to 1 and less than or equal to 5.

[0132] Specifically, with reference to FIG. 4, when performing 3-layer Laplacian pyramid decomposition on the visible light original image, the obtained 0th Laplacian layer L0, 1st Laplacian layer L1 and 2nd Laplacian layer L2 are respectively the 0th visible light detail layer, the 1st visible light detail layer and the 2nd visible light detail layer; when performing 3-layer Laplacian pyramid decomposition on the infrared original image, the obtained 0th Laplacian layer L0, 1st Laplacian layer L1 and 2nd Laplacian layer L2 are respectively the 0th infrared detail layer, the 1st infrared detail layer and the 2nd infrared detail layer.

[0133] Specifically, as shown in some embodiments of FIG. 4, the image processing method further comprises: performing image fusion on the 0th visible light detail layer and the 0th infrared detail layer to obtain a 0th detail fusion image F0 (as shown in image F0 of FIG. 4); performing image fusion on the 1st visible light detail layer and the 1st infrared detail layer to obtain a 1st detail fusion image F1 (as shown in image F1 of FIG. 4); performing image fusion on the 2nd visible light detail layer and the 2nd infrared detail layer to obtain a 2nd detail fusion image F2 (as shown in image F2 of FIG. 4).

[0134] As shown in FIG. 6, in some embodiments of the present application, the step S7 of performing image fusion on the a-th visible light detail layer and the a-th infrared detail layer to obtain a a-th detail fusion image comprises: a step S72 of performing image fusion on the a-th visible light detail layer and the a-th infrared detail layer according to a a-th detail fusion weight to obtain a a-th detail fusion image.

[0135] As shown in FIG. 6, in some embodiments, the step S7 of performing image fusion on the a-th visible light detail layer and the a-th infrared detail layer to obtain a a-th detail fusion image further comprises: a step S71 of obtaining a a-th detail fusion weight.

[0136] Referring to FIG. 7, FIG. 7 shows a flowchart of the step of obtaining a detail fusion weight in the image processing method of the present application.

[0137] In some embodiments of the present application, the step S71 of obtaining a a-th detail fusion weight comprises:

[0138] a step S711 of obtaining an activity of the a-th infrared detail layer according to the a-th infrared detail layer;

[0139] a step S712 of obtaining an activity of the a-th visible light detail layer according to the a-th visible light detail layer;

[0140] a step S713 of obtaining a a-th detail fusion weight according to the activity of the a-th infrared detail layer and the activity of the a-th visible light detail layer, and combining a a-th infrared noise reduction parameter and a a-th visible light noise reduction parameter.

[0141] For example, the filter kernel of the mean filter is a filter kernel of 1x5: In other embodiments, a larger filter kernel can also be selected when resources are sufficient.

[0142] In some embodiments, the step of obtaining an activity of a detail layer comprises performing mean filtering on an absolute value of the detail layer to obtain the activity of the detail layer. Specifically, in the step S711 of obtaining an activity of the a-th infrared detail layer according to the a-th infrared detail layer, the absolute value of the a-th infrared detail layer is subjected to mean filtering to obtain the activity act_ir of the a-th infrared detail layer: act_ir = blur(abs(L2i)); in the step S712 of obtaining an activity of the a-th visible light detail layer according to the a-th visible light detail layer, the absolute value of the a-th visible light detail layer is subjected to mean filtering to obtain the activity act_vi of the a-th visible light detail layer: act_vi = blur(abs(L2v)).

[0143] Step S713: In the step of obtaining the a-th detail fusion weight, the a-th infrared noise reduction parameter and the a-th visible light noise reduction parameter are suitable for adjusting the parameters of the respective weights of the a-th infrared detail layer and the a-th visible light detail layer respectively to suppress noise. In some embodiments, when the original image noise is large, adjusting the a-th infrared noise reduction parameter and the a-th visible light noise reduction parameter can help reduce noise.

[0144] As shown in FIG. 4, after obtaining the a-th detail fusion weight, the a-th visible light detail layer and the a-th infrared detail layer are fused according to the a-th detail fusion weight to obtain the a-th detail fusion image.

[0145] Specifically, after the step of obtaining the a-th detail fusion weight, the a-th detail layer fusion image Lar is calculated according to the detail fusion weight, each infrared pixel value in the infrared detail layer, and each visible light pixel value in the visible light detail layer. Lavi Lar = Lav * weight + Lai * (1 - weight Lavi weight), where Lar represents the a-th detail layer fusion image, weight represents the a-th detail fusion weight, Lav represents the a-th visible light detail layer, and Lai represents the a-th infrared detail layer. Lavi

[0146] In some embodiments as shown in FIG. 4, the second visible light detail layer is the second Laplacian layer L2 in the Laplacian pyramid obtained by performing 3-layer Laplacian pyramid decomposition on the visible light original image; and the second infrared detail layer is the second Laplacian layer L2 in the Laplacian pyramid obtained by performing 3-layer Laplacian pyramid decomposition on the infrared original image.

[0147] The second Laplacian layer L2 contains large edges in the image and relatively less noise. The weight can be calculated according to the activity, and then fused according to the weight (Weighted Average, WA).

[0148] According to the a-th infrared detail layer, the step of obtaining the activity of the a-th infrared detail layer is as follows: taking the absolute value of the second infrared detail layer L2i to obtain the second infrared detail layer abs(L2i); and performing mean filtering on the second infrared detail layer abs(L2i) using a 1x5 filter kernel to obtain the activity act_ir of the second infrared detail layer L2i: act_ir = blur(abs(L2i)), where blur is a filter function and abs is an absolute value function.

[0149] ​According to the a visible light detail layer, the step of obtaining the activity of the a visible light detail layer is as follows: taking the absolute value of the second visible light detail layer L2v to obtain the absolute value of the second visible light detail layer abs(L2v); performing mean filtering on the absolute value of the second visible light detail layer abs(L2v) by using a 1x5 filter kernel to obtain the activity act_v of the second visible light detail layer L2v: act_v = blur(abs(L2v)), blur is a filter function, and abs is an absolute value function.

[0150] It should be noted that the filter kernel of the 1x5 mean filtering is as follows: The filter kernel in the form of a line can save the design of the line buffer, and in the case of sufficient resources, a larger filter kernel can also be selected.

[0151] Then, the second detail fusion weight is calculated:

[0152] wherein weight L2vi represents the second detail fusion weight, k L2vi and k L2ir respectively represent the second visible light noise reduction parameter and the second infrared noise reduction parameter, and take integer values between 1 and 64, and the initial value is 1. When the noise of one of the second visible light detail layer and the second infrared detail layer is large, the second visible light noise reduction parameter k L2vi and the second infrared noise reduction parameter k L2ir can help reduce noise. Then, according to the detail fusion weight, each infrared pixel value in the infrared detail layer, and each visible light pixel value in the visible light detail layer, the second detail layer fusion image L2r = L2v*weight L2vi + L2i*(1-weight L2vi ) is calculated, L2r represents the second detail layer fusion image, weight L2vi represents the second detail fusion weight, L2v is the second visible light detail layer, and L2i is the second infrared detail layer.

[0153] Then, the first detail fusion weight is calculated:

[0154] wherein weight L1vi represents the first detail fusion weight, k L1vi and k L1ir respectively represent the first visible light noise reduction parameter and the first infrared noise reduction parameter, and take integer values between 1 and 64, and the initial value is 1. When the noise of one of the first visible light detail layer and the first infrared detail layer is large, the first visible light noise reduction parameter k L1vi and the first infrared noise reduction parameter kL1ir This can help with noise reduction. Next, based on the detail fusion weights, the infrared pixel values ​​within the infrared detail layer, and the visible light pixel values ​​within the visible light detail layer, the first detail layer fused image L1r = L1v * weight is calculated. L1vi +L1i*(1-weight L1vi L1r represents the fused image of the first detail layer, weight L1vi L1 represents the first detail fusion weight, L1v is the first visible light detail layer, and L1i is the first infrared detail layer.

[0155] Next, calculate the 0th detail fusion weight:

[0156] Where, weight L0vi Represented as the 0th detail fusion weight, k L0vi and k L0ir These represent the 0th visible light noise reduction parameter and the 0th infrared noise reduction parameter, respectively, with values ​​ranging from 1 to 64, and an initial value of 1. When the noise in either the 0th visible light detail layer or the 0th infrared detail layer image is high, the 0th visible light noise reduction parameter k is adjusted. L0vi And the 0th infrared noise reduction parameter k L0ir This can help with noise reduction. Next, based on the detail fusion weights, the infrared pixel values ​​within the infrared detail layer, and the visible light pixel values ​​within the visible light detail layer, the fused image L0r = L0v * weight of the 0th detail layer is calculated. L0vi +L0i*(1-weight L0vi L0r represents the fused image at the 0th detail layer, weight L0vi This is represented as the 0th detail fusion weight, L0v is the 0th visible light detail layer, and L0i is the 0th infrared detail layer.

[0157] In the above scheme, the visible light original image is decomposed into an N-layer pyramid to obtain a visible light detail layer, and the infrared original image is decomposed into an N-layer pyramid to obtain an infrared detail layer. The visible light detail layer and the infrared detail layer are then fused. Since the visible light detail layer and the infrared detail layer contain noise information, by setting different weight parameters and obtaining the fusion weight of the infrared detail layer and the fusion weight of the visible light detail layer based on the weight parameters and the activity, the influence of noise in the visible light detail layer and the infrared detail layer on the fusion effect can be effectively suppressed, and the fusion effect of the visible light detail layer and the infrared detail layer can be guaranteed.

[0158] It should be noted that in some embodiments of the present application, the first a detail fusion image is obtained by image fusion of the first a visible light detail layer and the first a infrared detail layer based on the first a detail fusion weight. However, this method is only an example.

[0159] In other embodiments of the present application, the first a detail fusion image is obtained by image fusion of the first a visible light detail layer and the first a infrared detail layer.

[0160] With reference to FIG. 1, in some embodiments of the present application, the image processing method further comprises: step 8, image restoration according to the base layer fusion image to obtain a gray-scale fusion image.

[0161] In some embodiments, step S1, the N-layer pyramid decomposition is performed on the visible light original image to obtain N visible light detail layers; step S2, the N-layer pyramid decomposition is performed on the infrared original image to obtain N infrared detail layers; the image processing method further comprises: step 7, image fusion of the first a visible light detail layer and the first a infrared detail layer to obtain the first a detail fusion image; and step 8, image restoration according to the base layer fusion image to obtain the gray-scale fusion image, wherein the image restoration is performed according to the base layer fusion image, the 0th detail fusion image, the a th detail fusion image, and the (N-1) th detail fusion image to obtain the gray-scale fusion image.

[0162] For example, in some embodiments as shown in FIG. 4, in the step of image restoration according to the base layer fusion image to obtain the gray-scale fusion image, the image restoration is performed according to the base layer fusion image, the 2nd detail fusion image, the 1st detail fusion image, and the 0th detail fusion image to obtain the gray-scale fusion image.

[0163] In some embodiments, step S8, the step of image restoration according to the base layer fusion image to obtain the gray-scale fusion image comprises: obtaining an nth restoration image GRn, 0≤n≤N; wherein the step of obtaining the nth restoration image GRn comprises: up-sampling the (n+1) th restoration image; and obtaining the nth restoration image GRn according to the sum of the up-sampled (n+1) th restoration image and the n th detail fusion image.

[0164] In some embodiments, step S8, the step of image restoration according to the base layer fusion image to obtain the gray-scale fusion image further comprises: sequentially obtaining an nth restoration image GRn, a (n+1) th restoration image GRn+1, and so on until a 0th restoration image GR0, wherein the 0th restoration image GR0 is the gray-scale fusion image.

[0165] It should be noted that in step S8, the step of performing image restoration on the basis layer fusion image to obtain a gray-scale fusion image, the basis layer fusion image is taken as the Nth restored image GRN.

[0166] In some embodiments as shown in FIG. 4 and FIG. 8, in step S8, the step of performing image restoration on the basis layer fusion image to obtain a gray-scale fusion image, the third layer restored image GR3, the second layer restored image GR2, the first layer restored image GR1 and the zeroth layer restored image GR0 obtained in sequence are respectively:

[0167] The basis layer fusion image F3 is taken as the Nth restored image GR3; the third restored image GR3 is up-sampled to obtain a sampled third restored image GR3↑, and the second restored image GR2 is obtained according to the sampled third restored image GR3↑ and L2; the second restored image GR2 is up-sampled to obtain a sampled second restored image GR2↑, and the first restored image GR1 is obtained according to the sampled second restored image GR2↑ and L1; the first restored image GR1 is up-sampled to obtain a sampled first restored image GR1↑, and the zeroth restored image GR0 is obtained according to the sampled first restored image GR1↑ and L0.

[0168] The zeroth restored image GR0 is a gray-scale fusion image.

[0169] Referring to FIG. 9, the effect of comparing the technical scheme of the present application with other technologies is compared, and the technical effect of the present application is discussed again. FIG. 9 shows an effect comparison diagram of fusion images obtained by using different fusion methods in the image processing method of the present application. In FIG. 9, (a) is a visible light original image, (b) is an infrared original image, (c) is a fusion image obtained by using a fusion method of the prior art (for example, average fusion), and (d) is a fusion image obtained by using the fusion method of the present application.

[0170] As can be seen, the image effect of the present scheme is that the fusion effect is like “injecting” an infrared thermal target into a visible light image, so that the details and contrast of the visible light image can be better preserved, and the infrared thermal target can be highlighted.

[0171] As shown in FIG. 10, (a) of FIG. 10 is a gray-scale fusion image obtained in some embodiments of the present application, that is, a basis layer fusion image obtained by performing image fusion on the visible light basis layer and the infrared basis layer according to the infrared pixel weight curve and the visible light pixel weight curve, and then obtaining a gray-scale fusion image; (b) of FIG. 10 is a gray-scale fusion image obtained by performing average fusion on the visible light basis layer and the infrared basis layer.

[0172] It can be known from comparing the (a) of Fig. 10 and the (b) of Fig. 10 that the (a) of Fig. 10 highlights the infrared thermal target, for example, the water boiler in the (a) of Fig. 10 can be clearly distinguished from the background, and other details in the (a) of Fig. 10 can also be reserved, while in the (b) of Fig. 10, the traditional average fusion mode is adopted, and the contrast of the target image is insufficient, especially for the target image with opposite light and shade in the visible light base layer and the infrared base layer, for example, the black water boiler in the (b) of Fig. 10 is placed in front of the white background, and due to its high temperature, it is brighter than the background in the infrared image, and after the traditional average fusion mode is adopted, the contrast of the fusion image is low, and the thermal target cannot be highlighted.

[0173] Therefore, by using the above scheme, the visible light original image is subjected to N-layer pyramid decomposition to obtain a visible light base layer; the infrared original image is subjected to N-layer pyramid decomposition to obtain an infrared base layer, and the visible light base layer and the infrared base layer are fused according to the infrared pixel weight curve and the visible light pixel weight curve, so that the display weight of different pixel values of the infrared base layer in the base layer fusion image is high or low, thereby achieving the effect of “injecting” the infrared base layer into the visible light base layer, which can effectively improve the highlighting degree of the infrared target after fusion; and since only structural information is included in the visible light base layer and the infrared base layer, the highlighting display of the infrared base layer in the base layer fusion image can effectively avoid introducing noise information, and can effectively guarantee the fusion effect of the visible light base layer and the infrared base layer.

[0174] It should be noted that some of the above embodiments are black and white fusion algorithms for the visible light original image and the infrared original image, that is, the obtained fusion image is a gray-scale fusion image, that is, the gray-scale fusion image is black and white. In other embodiments of the present application, colorization can also be performed based on the visible light original image and the infrared original image to obtain a color image.

[0175] Referring to Fig. 11, a flow diagram of another embodiment of the image processing method of the present application is shown.

[0176] The same as the foregoing embodiments, the present application will not be described here. The difference from the foregoing embodiments is that in some embodiments of the present application, colorization is performed based on the visible light original image and the infrared original image to obtain a color image.

[0177] Step S1, N-layer pyramid decomposition is performed on the visible light original image to obtain a visible light base layer; then, step S2, N-layer pyramid decomposition is performed on the infrared original image to obtain an infrared base layer; then, step S3, image fusion is performed on the visible light base layer and the infrared base layer according to an infrared pixel weight curve and a visible light pixel weight curve to obtain a base layer fused image; then, step S8, image restoration is performed according to the base layer fused image to obtain a gray-scale fused image.

[0178] In the process of step S1, N-layer pyramid decomposition is performed on the visible light original image to obtain a visible light base layer, in the step of performing Laplacian pyramid decomposition on the visible light original image for the jth time, an m-1 visible Gaussian layer is obtained; in the process of step S2, N-layer pyramid decomposition is performed on the infrared original image to obtain an infrared base layer, in the step of performing Laplacian pyramid decomposition on the infrared original image for the jth time, an m-1 infrared Gaussian layer is obtained.

[0179] As shown in FIG. 11, in some embodiments of the present application, the image processing method further comprises: step S9, obtaining two chroma layers according to the m-1 visible Gaussian layer and the m-1 infrared Gaussian layer; step S10, performing m-1 times of up-sampling on the two chroma layers respectively to obtain two chroma channel images; and step S11, obtaining a color image according to the two chroma channel images and the gray-scale fused image.

[0180] In some embodiments, in the step S9 of obtaining two chroma layers according to the m-1 visible Gaussian layer and the m-1 infrared Gaussian layer, two chroma layers are obtained according to the N-1 visible Gaussian layer and the N-1 infrared Gaussian layer. The N-1 visible Gaussian layer and the N-1 infrared Gaussian layer are Gaussian layers obtained in the last time of performing the image decomposition step. The noise in the Gaussian layer obtained in the last time of performing the image decomposition step is less, so that the noise in the obtained chroma layer is less, and storage resources are saved.

[0181] In some specific embodiments, the image processing method further comprises: setting a pseudo-color parameter; and in the step S9 of obtaining two chroma layers according to the m-1 visible Gaussian layer and the m-1 infrared Gaussian layer, two chroma layers are obtained according to the m-1 visible Gaussian layer and the m-1 infrared Gaussian layer in combination with the pseudo-color parameter.

[0182] For example, in the step of setting a pseudo-color parameter, the pseudo-color parameter can be set according to a color style.

[0183] Specifically, the step of setting the pseudo-color parameters comprises: setting pseudo-color parameters corresponding to the color difference layer and the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer respectively; and setting another pseudo-color parameter corresponding to the color difference layer and the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer respectively.

[0184] In the step of obtaining two color difference layers according to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer in combination with the pseudo-color parameters, two color difference layers are obtained according to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer in combination with the corresponding pseudo-color parameters. Specifically:

[0185] U_L m-1 = G m-1 e_vi * q1 - G m-1 e_ir * q2;

[0186] V_L m-1 = G m-1 e_vi * q3 - G m-1 e_ir * q4.

[0187] wherein U_L m-1 is a U channel color difference layer corresponding to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer, V_L m-1 is a V channel color difference layer corresponding to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer, G m-1 e_vi is the m-1th visible Gaussian layer, G m-1 e_ir is the m-1th infrared Gaussian layer, q1 is a pseudo-color parameter corresponding to the U channel color difference layer and the m-1th visible Gaussian layer, q2 is a pseudo-color parameter corresponding to the U channel color difference layer and the m-1th infrared Gaussian layer, q3 is a pseudo-color parameter corresponding to the V channel color difference layer and the m-1th visible Gaussian layer, and q4 is a pseudo-color parameter corresponding to the V channel color difference layer and the m-1th infrared Gaussian layer. In some embodiments of the example, in the step S1 of performing N-layer pyramid decomposition on the visible light original image, 3-layer pyramid decomposition is performed on the visible light original image; in the step S1 of performing N-layer pyramid decomposition on the infrared original image, 3-layer pyramid decomposition is performed on the infrared original image; and in the step of obtaining two color difference layers, the obtained color difference layers are respectively:

[0188] U_L2 = G2e_vi * q1 - G2e_ir * q2;

[0189] V_L2 = G2e_vi * q3 - G2e_ir * q4.

[0190] U_L2=q1*G2e_vi+q2*G2e_ir, V_L2=q3*G2e_vi+q4*G2e_ir, wherein U_L2 is a U channel chroma layer corresponding to the 2nd visible Gaussian layer and the 2nd infrared Gaussian layer, V_L2 is a V channel chroma layer corresponding to the 2nd visible Gaussian layer and the 2nd infrared Gaussian layer, G2e_vi is the 2nd visible Gaussian layer, G2e_ir is the 2nd infrared Gaussian layer, q1 is a pseudo-color parameter corresponding to the U channel chroma layer and the 2nd visible Gaussian layer, q2 is a pseudo-color parameter corresponding to the U channel chroma layer and the 2nd infrared Gaussian layer, q3 is a pseudo-color parameter corresponding to the V channel chroma layer and the 2nd visible Gaussian layer, and q4 is a pseudo-color parameter corresponding to the V channel chroma layer and the 2nd infrared Gaussian layer.

[0191] In some embodiments, step S10, 2 chroma layers are respectively subjected to m-1 times of up-sampling to obtain 2 chroma channel images.

[0192] Specifically, a U channel chroma layer U_L m-1 corresponding to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer is up-sampled to obtain U_L m-2 = sample_up(U_L m-1 ), wherein U_L m-2 is a U channel chroma layer corresponding to the m-2th visible Gaussian layer and the m-2th infrared Gaussian layer, and sample_up is an up-sampling process.

[0193] Specifically, a V channel chroma layer V_L m-1 corresponding to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer is up-sampled to obtain V_L m-2 = sample_up(V_L m-1 ), wherein V_L m-2 is a V channel chroma layer corresponding to the m-2th visible Gaussian layer and the m-2th infrared Gaussian layer, and sample_up is an up-sampling process.

[0194] The up-sampling results are taken as two chroma channel images, i.e., a U chroma channel image U L m-2out and a V chroma channel image V L m-2out , i.e., U L m-2out = U_L m-2 , and V L0out = V_L m-2 .

[0195] In other embodiments, after obtaining the U channel chroma layer U_L m-2 corresponding to the m-2th visible Gaussian layer and the m-2th infrared Gaussian layer, the U channel chroma layer U_L m-2up-sampling until a U channel chroma layer U_L0 corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer is obtained, U_L0 = sample_up(U_L1), wherein the 0th visible Gaussian layer and the 0th infrared Gaussian layer are the Gaussian layers obtained in the process of performing the image decomposition step for the first time, and U_L1 is a U channel chroma layer corresponding to the 1st visible Gaussian layer and the 1st infrared Gaussian layer.

[0196] In some other embodiments, after obtaining a V channel chroma layer V_L m-2 then, continue to up-sample the V channel chroma layer V_L m-2 up-sampling until a V channel chroma layer V_L0 corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer is obtained, V_L0 = sample_up(V_L1), wherein the 0th visible Gaussian layer and the 0th infrared Gaussian layer are the Gaussian layers obtained in the process of performing the image decomposition step for the first time, and V_L1 is a V channel chroma layer corresponding to the 1st visible Gaussian layer and the 1st infrared Gaussian layer.

[0197] The up-sampling results are taken as two chroma channel images, which are respectively a U chroma channel image U L0out and a V chroma channel image V L0out , i.e., U L0out = U_L0, V L0out = V_L0.

[0198] In some embodiments of the example, in the step S1 of performing N-layer pyramid decomposition on the visible light original image, 3-layer pyramid decomposition is performed on the visible light original image; in the step S1 of performing N-layer pyramid decomposition on the infrared original image, 3-layer pyramid decomposition is performed on the infrared original image; and in the step S10 of up-sampling the two chroma layers respectively for m-1 times to obtain two chroma channel images, the two chroma layers are up-sampled respectively for 2 times to obtain two chroma channel images.

[0199] Upsampling is performed on the U-channel chroma layer U_L2 corresponding to the second visible Gaussian layer and the second infrared Gaussian layer to obtain the U-channel chroma image U_L1 corresponding to the first visible Gaussian layer and the first infrared Gaussian layer; upsampling is performed on the U-channel chroma image U_L1 corresponding to the first visible Gaussian layer and the first infrared Gaussian layer to obtain the U-channel chroma image U_L0 corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer; upsampling is performed on the V-channel chroma image V_L2 corresponding to the second visible Gaussian layer and the second infrared Gaussian layer to obtain the V-channel chroma image V_L1 corresponding to the first visible Gaussian layer and the first infrared Gaussian layer; upsampling is performed on the V-channel chroma image V_L1 corresponding to the first visible Gaussian layer and the first infrared Gaussian layer to obtain the V-channel chroma image V_L0 corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer. The specific upsampling process is as follows:

[0200] U_L1 = sample_up(U_L2);

[0201] U_L0 = sample_up(U_L1);

[0202] V_L1 = sample_up(V_L2);

[0203] V_L0 = sample_up(V_L1).

[0204] Wherein, U_L2 is the U-channel chroma layer corresponding to the second visible Gaussian layer and the second infrared Gaussian layer, U_L1 is the U-channel chroma channel image corresponding to the first visible Gaussian layer and the first infrared Gaussian layer, U_L0 is the U-channel chroma channel image corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer, V_L1 is the V-channel chroma channel image corresponding to the first visible Gaussian layer and the first infrared Gaussian layer, V_L0 is the V-channel chroma channel image corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer, and sample_up is the upsampling process.

[0205] The above sampling results are used as two chroma channel images, namely: U chroma channel image U L0out and V chroma channel image V L0out , that is U L0out =U_L0,V L0out =V_L0.

[0206] In some embodiments, step S11 involves obtaining a color image based on the two chroma channel images and the grayscale fusion image.

[0207] Specifically, the image processing method further comprises: taking the gray-scale fused image as a luminance channel image; taking the above sampling result as two chrominance channel images; and obtaining, according to the two chrominance channel images and the gray-scale fused image, a color image in step S11, obtaining, according to the two chrominance channel images and the luminance channel image, a color image.

[0208] In some embodiments as shown in Fig. 4, in step S1, the visible light original image is subjected to 3-layer pyramid decomposition in the step of performing N-layer pyramid decomposition on the visible light original image; in step S1, the infrared original image is subjected to 3-layer pyramid decomposition in the step of performing N-layer pyramid decomposition on the infrared original image; and in step S11, according to the two chrominance channel images and the gray-scale fused image, a color image is obtained in that U L0out and V chrominance channel images U L0out and a luminance channel image Y L0out are obtained, and a color image in YUV space is obtained.

[0209] Specifically, in step S8, after the base layer fused image is subjected to image restoration, a 0th layer restored image GR0 is obtained, wherein the 0th layer restored image GR0 is the gray-scale fused image, i.e., Y L0out = GR0.

[0210] Referring to Fig. 4 and Fig. 12, compared with the gray-scale fused image in Fig. 4, (a) of Fig. 12, (b) of Fig. 12 and (c) of Fig. 12 are all color images, which expands the application scenarios of image fusion, and it can be known from (a) of Fig. 12, (b) of Fig. 12 and (c) of Fig. 12 that different color styles of color images are obtained through adjustment of pseudo-color parameters, so that the obtained color images have a natural feeling.

[0211] Referring to Fig. 13, a flow diagram of another embodiment of the image processing method of the present application is shown.

[0212] The same as the foregoing embodiments, the present application will not be described here. The difference from the foregoing embodiments is that in some embodiments of the present application, the image processing method can further perform clipping operation on the obtained detail layer to limit the size of the pixel value of the obtained detail layer, so as to alleviate the halo effect.

[0213] The halo effect (Halo Effect) represents an optical defect on an image; it usually appears at the edge of an image with strong changes in brightness and darkness.

[0214] In some embodiments of the present application, the visible light original image is subjected to N-layer pyramid decomposition, and N-layer visible light detail layers are obtained; the infrared original image is subjected to N-layer pyramid decomposition, and N-layer infrared detail layers are obtained; wherein N is greater than or equal to 1; the image processing method further comprises: in step S7, the a-th visible light detail layer and the a-th infrared detail layer are subjected to image fusion, and the a-th visible light detail layer and the a-th infrared detail layer are subjected to clipping operation respectively; in step S7, the a-th detail fusion image is obtained; in the step of obtaining the a-th detail fusion image, the a-th visible light detail layer subjected to the clipping operation and the a-th infrared detail layer subjected to the clipping operation are subjected to image fusion, and the a-th detail fusion image is obtained.

[0215] As shown in FIG. 13, in some embodiments of the present application, the image processing method comprises: step S1: the visible light original image is subjected to N-layer pyramid decomposition, and N-layer visible light detail layers are obtained; then, step S110: the a-th visible light detail layer is subjected to clipping operation; step S2: the infrared original image is subjected to N-layer pyramid decomposition, and N-layer infrared detail layers are obtained; then, step S210: the a-th infrared detail layer is subjected to clipping operation; then, step S220: the a-th visible light detail layer subjected to the clipping operation and the a-th infrared detail layer subjected to the clipping operation are subjected to image fusion, and the a-th detail fusion image is obtained.

[0216] Specifically, the clipping operation is to set the upper threshold and the lower threshold of the pixel value of the visible light detail layer, and set the upper threshold and the lower threshold of the pixel value of the infrared detail layer. In some embodiments, the pixel value range of the visible light detail layer is [-256, 255]; the pixel value range of the infrared detail layer is [-256, 255].

[0217] In some embodiments of the example, with reference to FIG. 4, in step S1, the visible light original image is subjected to 3-layer pyramid decomposition; in step S1, the infrared original image is subjected to 3-layer pyramid decomposition; the 0-th Laplace layer L0, the 1-st Laplace layer L1, and the 2-nd Laplace layer L2 are obtained; the Laplace layers in the left Laplace pyramid in FIG. 4 correspond to the 0-th visible light detail layer, the 1-st visible light detail layer, and the 2-nd visible light detail layer; the Laplace layers in the right Laplace pyramid in FIG. 4 correspond to the 0-th visible light detail layer, the 1-st visible light detail layer, and the 2-nd visible light detail layer; the upper threshold and the lower threshold of the pixel value of the 0-th visible light detail layer are set; the upper threshold and the lower threshold of the pixel value of the 1-st visible light detail layer are set; the upper threshold and the lower threshold of the pixel value of the 2-nd visible light detail layer are set; the upper threshold and the lower threshold of the pixel value of the 0-th infrared detail layer are set; the upper threshold and the lower threshold of the pixel value of the 1-st infrared detail layer are set; the upper threshold and the lower threshold of the pixel value of the 2-nd infrared detail layer are set.

[0218] As shown in FIG. 14, where FIG. 14(a) is a gray-scale fused image obtained without pinch-off operation in some embodiments of the present application, and FIG. 14(b) is a gray-scale fused image obtained with pinch-off operation in some embodiments of the present application.

[0219] As can be seen by comparing FIG. 14(a) and FIG. 14(b), the gray-scale fused image obtained after pinch-off operation on the visible light detail layer and the infrared detail layer can suppress the influence of halo effect and reduce resource occupation.

[0220] Referring to FIG. 15, a schematic diagram of another embodiment of the image processing method of the present application is shown.

[0221] The same as the foregoing embodiments, the present application will not be described here. The difference from the foregoing embodiments is that the number of endpoints of the infrared pixel weight curve can be greater than 4 in some embodiments of the present application.

[0222] As shown in FIG. 15, the acquisition of the infrared pixel weight curve is specifically connecting the first endpoint A1 and the second endpoint A2, and the second endpoint A2 and the third endpoint A3 to obtain the infrared pixel weight curve of the cold target region.

[0223] Connecting the third endpoint A3 and the fourth endpoint A4 to obtain the infrared pixel weight curve of the background target region.

[0224] Connecting the fourth endpoint A4 and the fifth endpoint A5, and the fifth endpoint A5 and the sixth endpoint A6 to obtain the infrared pixel weight curve of the hot target region.

[0225] Referring to FIG. 16, where FIG. 16(a) is a gray-scale fused image obtained according to the infrared pixel weight curve determined by four endpoints, and FIG. 16(b) is a gray-scale fused image obtained according to the infrared pixel weight curve determined by six endpoints.

[0226] As can be seen by comparing FIG. 16(a) and FIG. 16(b), the method of determining the infrared pixel weight curve according to six endpoints not only improves the fusion ratio of the infrared base layer, but also makes the infrared pixel weight curve adjustment more fine, and the infrared hot target in the obtained gray-scale fused image is more prominent, for example, the water kettle with higher temperature in FIG. 16(b) is more prominent in the fused image.

[0227] Correspondingly, the present application also provides an image processing device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the program, the processor executes the steps of the image processing method described above.

[0228] In the image processing device, the processor executes the computer program stored on the memory to execute the steps of the image processing method of the present application, and the specific technical solutions of the image processing device can refer to the embodiments of the image processing method described above, which will not be repeated here.

[0229] Correspondingly, the present application also provides a computer readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, and stores computer instructions thereon, and when the computer instructions are executed, the steps of the image processing method described above are executed.

[0230] The computer readable storage medium stores computer instructions which, when executed, execute the steps of the image processing method of the present application, and the specific technical solutions of the computer readable storage medium can refer to the embodiments of the image processing method described above, which will not be repeated here.

[0231] In summary, by performing N-layer pyramid decomposition on the visible light original image, a visible light base layer is obtained; by performing N-layer pyramid decomposition on the infrared original image, an infrared base layer is obtained, and the visible light base layer and the infrared base layer are fused according to the infrared pixel weight curve and the visible light pixel weight curve, so that the display weight of different pixel values of the infrared base layer in the base layer fusion image is high and low, thereby achieving the effect of "injecting" the infrared base layer into the visible light base layer, which can effectively improve the prominence of the infrared target after fusion. Moreover, since only structural information is included in the visible light base layer and the infrared base layer, the prominent display of the infrared base layer in the base layer fusion image can effectively avoid introducing noise information and effectively ensure the fusion effect of the visible light base layer and the infrared base layer.

[0232] Moreover, by setting a background target field, a hot target field and a cold target field in the range of the infrared base layer pixel value, and the infrared pixel weight corresponding to the infrared pixel value in different target fields being different, the display weight of the infrared pixel value in different target fields in the base layer fusion image is high and low, thereby ensuring that the infrared base layer can be prominently displayed in the base layer fusion image. Moreover, the infrared pixel weight corresponding to the infrared pixel value in the cold target field and the infrared pixel weight corresponding to the infrared pixel value in the hot target field are both greater than the infrared pixel weight corresponding to the infrared pixel value in the background target field, which can effectively improve the contrast and naturalness of the fused image and effectively improve the fusion effect.

[0233] In addition, the present application sets the initial weight value corresponding to the minimum infrared pixel value, the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the second infrared pixel value, and the initial infrared pixel weight corresponding to the maximum infrared pixel value, that is, sets different initial infrared pixel weights, and obtains the infrared pixel weight curve in different target fields according to the different initial infrared pixel weights, and then the infrared weight curve can be obtained by adjusting the infrared pixel weight corresponding to the different infrared pixel value, so as to realize the optimization of the infrared weight curve, and make the infrared base layer highlight in the base layer fusion image.

[0234] In addition, the technical scheme of the present application obtains the visible light detail layer and the infrared detail layer by performing N-layer image decomposition on the visible light original image and the infrared original image, and performs image fusion on the a-th visible light detail layer and the a-th infrared detail layer according to the a-th detail fusion weight. Since the visible light detail layer and the infrared detail layer include noise information, and the calculation of the detail fusion weight can adjust the noise in the visible light detail layer and the infrared detail layer, the image fusion using the detail fusion weight can reduce the noise in the visible light detail layer and the infrared detail layer, thereby ensuring the fusion effect of the visible light detail layer and the infrared detail layer.

[0235] In addition, the technical scheme of the present application performs clipping operation on the infrared detail layer and the visible light detail layer, that is, threshold limiting processing on the infrared detail layer and the visible light detail layer, so that the pixel values in the infrared detail layer and the visible light detail layer are limited, resource occupation is reduced, and the generation of the halo effect is inhibited.

[0236] In addition, the technical scheme of the present application obtains two chroma channel images according to the visible Gaussian layer and the infrared Gaussian layer obtained after N-layer pyramid decomposition, and performs color component processing according to the two chroma channel images and the gray-scale fusion image to obtain a color output image, thereby expanding the application scenario of image fusion.

[0237] Although the present application is disclosed as above, the present application is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, and therefore the protection scope of the present application should be subject to the scope defined by the claims.

Claims

An image processing method, characterized by, The method comprises the following steps: performing N-layer pyramid decomposition on a visible light original image to obtain a visible light base layer; performing N-layer pyramid decomposition on an infrared original image to obtain an infrared base layer, wherein N is a natural number; performing image fusion on the visible light base layer and the infrared base layer according to an infrared pixel weight curve and a visible light pixel weight curve to obtain a base layer fusion image. The image processing method of claim 1, wherein, The image processing method further comprises the step of setting an infrared pixel weight curve; The step of setting the infrared pixel weight curve comprises the following steps: setting a background target region, a hot target region and a cold target region in the range of the pixel values of the infrared base layer, wherein the infrared pixel values in the background target region are between the infrared pixel values in the hot target region and the infrared pixel values in the cold target region; respectively setting the infrared pixel weight curve of the background target region, the infrared pixel weight curve of the hot target region and the infrared pixel weight curve of the cold target region, wherein the infrared pixel weight corresponding to the infrared pixel value in the background target region is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the hot target region; and the infrared pixel weight corresponding to the infrared pixel value in the background target region is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the cold target region. The image processing method of claim 2, wherein, In the step of setting the background target region, the hot target region and the cold target region in the range of the pixel values of the infrared base layer, the average value of the pixel values of the infrared base layer is in the background target region. The image processing method of claim 2, wherein, respectively setting an initial infrared pixel weight corresponding to a minimum infrared pixel value, an initial infrared pixel weight corresponding to a first infrared pixel value, an initial infrared pixel weight corresponding to a second infrared pixel value and an initial infrared pixel weight corresponding to a maximum infrared pixel value, wherein the minimum infrared pixel value, the first infrared pixel value, the second infrared pixel value and the maximum infrared pixel value increase in sequence, the initial infrared pixel weight corresponding to the minimum infrared pixel value is greater than or equal to the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the maximum infrared pixel value is greater than or equal to the initial infrared pixel weight corresponding to the second infrared pixel value, and the minimum infrared pixel value and the first infrared pixel value are in the cold target region, the first infrared pixel value and the second infrared pixel value are in the background target region, and the second infrared pixel value and the maximum infrared pixel value are in the hot target region; obtaining the infrared pixel weight curve of the cold target region according to the initial infrared pixel weight corresponding to the minimum infrared pixel value and the initial infrared pixel weight corresponding to the first infrared pixel value; obtaining the infrared pixel weight curve of the background target region according to the initial infrared pixel weight corresponding to the first infrared pixel value and the initial infrared pixel weight corresponding to the second infrared pixel value; obtaining the infrared pixel weight curve of the hot target region according to the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value. The step of obtaining the base layer fusion image comprises the following steps: The image processing method of claim 2, wherein, ​ According to the infrared pixel weight curve, the visible light pixel weight curve, each visible light pixel value in the visible light base layer and each infrared pixel value in the infrared base layer, a base layer fusion image Gnr = w_vis*Gnv + w_ir*Gni is calculated, Gnr represents the base layer fusion image, w_vis represents the visible light pixel weight, w_ir represents the infrared pixel weight, Gnv represents the visible light pixel value, Gni represents the infrared pixel value, and n is a natural number. The image processing method of claim 1, wherein, The visible light original image is subjected to N-layer pyramid decomposition, and N visible light detail layers are obtained. The infrared original image is subjected to N-layer pyramid decomposition, and N infrared detail layers are obtained. N is greater than or equal to 1. The image processing method further comprises: The a-th visible light detail layer and the a-th infrared detail layer are subjected to image fusion to obtain an a-th detail fusion image, wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1. In the step of subjecting the a-th visible light detail layer and the a-th infrared detail layer to image fusion to obtain the a-th detail fusion image, the a-th visible light detail layer and the a-th infrared detail layer are subjected to average fusion to obtain the a-th detail fusion image. The image processing method of claim 6, wherein, In the step of subjecting the a-th visible light detail layer and the a-th infrared detail layer to image fusion to obtain the a-th detail fusion image, the a-th visible light detail layer and the a-th infrared detail layer are subjected to image fusion according to an a-th detail fusion weight to obtain the a-th detail fusion image. The image processing method of claim 6, wherein, The step of obtaining the a-th detail fusion image comprises: The image processing method of claim 8, wherein, In the step of subjecting the a-th visible light detail layer and the a-th infrared detail layer to image fusion to obtain the a-th detail fusion image, an a-th detail fusion weight is obtained. According to the a-th detail fusion weight, each infrared pixel value in the infrared detail layer, and each visible light pixel value in the visible light detail layer, a detail layer fusion image Lar = Lav*weight Lavi + Lai*(1-weight Lavi ) is calculated, where Lar represents the a-th detail layer fusion image, weight Lavi represents the a-th detail fusion weight, Lav is the a-th visible light detail layer, and Lai is the a-th infrared detail layer. The image processing method of claim 8, wherein, The step of obtaining the a-th detail fusion weight comprises: The image processing method of claim 10, wherein, According to the a-th infrared detail layer, an activity of the a-th infrared detail layer is obtained. According to the a-th visible light detail layer, an activity of the a-th visible light detail layer is obtained. According to the activity of the a-th infrared detail layer and the activity of the a-th visible light detail layer, an a-th layer detail fusion weight is obtained in combination with an a-th infrared noise reduction parameter and an a-th visible light noise reduction parameter. Further comprising: The image processing method of claim 6, wherein, Before the step of subjecting the a-th visible light detail layer and the a-th infrared detail layer to image fusion to obtain the a-th detail fusion image, the a-th visible light detail layer and the a-th infrared detail layer are subjected to clipping operation respectively; In the step of obtaining the a-th detail fusion image, the a-th visible light detail layer subjected to the clipping operation and the a-th infrared detail layer subjected to the clipping operation are subjected to image fusion to obtain the a-th detail fusion image. The image processing method further comprises: performing image restoration according to the base layer fusion image to obtain a gray-scale fusion image. The image processing method of claim 1, wherein, The visible light original image is subjected to N-layer pyramid decomposition, and N visible light detail layers are obtained. The image processing method of claim 13, wherein, The infrared original image is subjected to N-layer pyramid decomposition, and N infrared detail layers are obtained. N is greater than or equal to 1. The image processing method further comprises: subjecting the a-th visible light detail layer and the a-th infrared detail layer to image fusion to obtain an a-th detail fusion image, wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1. ​ In the step of obtaining the gray fused image, image restoration is performed according to the base layer fused image, the 0th detail fused image, the ath detail fused image, and the N-1th detail fused image to obtain the gray fused image. The image processing method of claim 1, wherein, In at least one of the step of performing N-layer pyramid decomposition on the visible light original image and the step of performing N-layer pyramid decomposition on the infrared original image, N-layer Laplacian pyramid decomposition is performed on the to-be-processed image. The to-be-processed image is one of the visible light original image and the infrared original image. The image processing method of claim 15, wherein, The step of performing N-layer Laplacian pyramid decomposition comprises: performing the Laplacian pyramid decomposition step at least once, and performing the Laplacian pyramid decomposition step for the jth time to obtain a (j-1)th Laplacian layer, where j is a positive integer greater than or equal to 0 and less than or equal to N. The step of performing the Laplacian pyramid decomposition step for the mth time comprises: performing Gaussian filtering on the (m-1)th layer image; performing 1 / 2 down-sampling on the Gaussian-filtered (m-1)th layer image to obtain a sampled and filtered mth layer image; performing up-sampling on the mth layer image to obtain an (m-1)th Gaussian layer; obtaining an (m-1)th Laplacian layer according to the (m-1)th Gaussian layer and the (m-1)th layer image, where m is a positive integer greater than or equal to 2 and less than or equal to N. The image processing method of claim 16, wherein, In the step of performing the Laplacian pyramid decomposition step for the first time, Gaussian filtering is performed on the to-be-processed image to obtain a 0th Laplacian layer. The image processing method of claim 16, wherein, The image processing method further comprises: performing image restoration according to the base layer fused image to obtain a gray fused image. obtaining two chroma layers according to the (m-1)th visible Gaussian layer and the (m-1)th infrared Gaussian layer; performing m-1 times of up-sampling on the two chroma layers respectively to obtain two chroma channel images; obtaining a color image according to the two chroma channel images and the gray fused image. In the step of obtaining two chroma layers according to the (m-1)th visible Gaussian layer and the (m-1)th infrared Gaussian layer, two chroma layers are obtained according to the (N-1)th visible Gaussian layer and the (N-1)th infrared Gaussian layer. The image processing method of claim 18, wherein, The image processing device comprises: a memory and a processor, the memory stores a computer program executable on the processor, and the processor executes the steps of the image processing method according to any one of claims 1 to 19 when executing the program. An image processing apparatus characterized by comprising: The computer readable storage medium is a non-volatile storage medium or a non-transient storage medium, and computer instructions are stored thereon, and the computer instructions execute the steps of the image processing method according to any one of claims 1 to 19 when running. ​ A computer-readable storage medium, characterized by ​

Images