Image processing method and apparatus, and computer readable storage medium

Abstract

An image processing method and apparatus, and a computer readable storage medium. The image processing method comprises: performing N-level pyramid decomposition on a visible light original image to obtain N visible light detail layers and a visible light base layer (S1); and performing N-level pyramid decomposition on an infrared original image to obtain N infrared detail layers and an infrared base layer (S2); and on the basis of the a-th detail fusion weight, performing image fusion on the a-th visible light detail layer and the a-th infrared detail layer to obtain the a-th detail fusion image (S3). The visible light original image and the infrared original image are subjected to N-level image decomposition to obtain the visible light detail layers and the infrared detail layers, and the visible light detail layers and the infrared detail layers are subjected to image fusion on the basis of the detail fusion weight. Since the visible light detail layers and the infrared detail layers comprise noise information, performing image fusion on the basis of the fusion weight can reduce the noise in the visible light detail layers and the infrared detail layers, thereby ensuring the fusion effect of the visible light detail layers and the infrared detail layers.

Description

Image processing method, image processing device thereof, and computer readable storage medium

[0001] The present application claims priority to the Chinese patent application No. 2024111702162, filed on August 23, 2024, and entitled "Image processing method, image processing device thereof, and computer readable storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of image acquisition, in particular to an image processing method, an image processing device thereof, and a computer readable storage medium. BACKGROUND

[0003] Infrared imaging is not affected by the intensity of ambient light and can work in a weak light environment, so infrared images are widely used in target recognition and detection fields, but the texture details are relatively blurred; visible light images can provide rich texture details. After fusing the visible light image and the infrared image, the fusion image can have the advantages of both.

[0004] However, the fusion effect of the prior art visible light image and infrared image is not good, for example, the infrared target is not prominent enough, the image lacks naturalness, and the noise is large.

[0005] SUMMARY

[0006] The technical problem solved by the present application is how to improve the fusion effect of the visible light image and the infrared image.

[0007] To solve the above technical problem, an image processing method is provided, comprising: performing N-layer pyramid decomposition on a visible light original image to obtain N visible light detail layers; performing N-layer pyramid decomposition on an infrared original image to obtain N infrared detail layers, wherein N is greater than or equal to 1; performing image fusion on the ath visible light detail layer and the ath infrared detail layer according to the ath detail fusion weight to obtain the ath detail fusion image; wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1.

[0008] Optionally, the step of obtaining the ath detail fusion image comprises: calculating a detail layer fusion image Lar=Lav*weight Lavi +Lai*(1-weight Lavi ) according to the ath detail fusion weight, each infrared pixel value in the infrared detail layer, and each visible light pixel value in the visible light detail layer, wherein Lar represents the detail layer fusion image, weight Lavi represents the ath detail fusion weight, Lav is the ath visible light detail layer, and Lai is the ath infrared detail layer.

[0009] Optionally, the step of obtaining the a-th detail fusion image by fusing the a-th visible light detail layer and the a-th infrared detail layer comprises: obtaining a-th detail fusion weight.

[0010] Optionally, the step of obtaining the a-th detail fusion weight comprises: obtaining an activity of the a-th infrared detail layer according to the a-th infrared detail layer; obtaining an activity of the a-th visible light detail layer according to the a-th visible light detail layer; obtaining the a-th detail fusion weight according to the activity of the a-th infrared detail layer, the activity of the a-th visible light detail layer, the a-th infrared denoising parameter and the a-th visible light denoising parameter.

[0011] Optionally, the image processing method further comprises: before obtaining the a-th detail fusion image by fusing the a-th visible light detail layer and the a-th infrared detail layer, respectively performing a clipping operation on the a-th visible light detail layer and the a-th infrared detail layer; in the step of obtaining the a-th detail fusion image, fusing the a-th visible light detail layer after the clipping operation and the a-th infrared detail layer after the clipping operation to obtain the a-th detail fusion image.

[0012] Optionally, the step of performing N-layer pyramid decomposition on the visible light original image further obtains a visible light base layer; the step of performing N-layer pyramid decomposition on the infrared original image further obtains an infrared base layer, wherein N is a natural number; the image processing method further comprises: fusing the visible light base layer and the infrared base layer to obtain a base layer fusion image.

[0013] Optionally, in the step of fusing the visible light base layer and the infrared base layer to obtain the base layer fusion image, the visible light base layer and the infrared base layer are averaged to obtain the base layer fusion image.

[0014] Optionally, in the step of fusing the visible light base layer and the infrared base layer to obtain the base layer fusion image, the visible light base layer and the infrared base layer are fused according to an infrared pixel weight curve and a visible light pixel weight curve to obtain the base layer fusion image.

[0015] Optionally, the image processing method further comprises: setting an infrared pixel weight curve; the step of setting the infrared pixel weight curve comprises: setting a background target region, a hot target region and a cold target region in the infrared base layer pixel value range, wherein the infrared pixel value in the background target region is between the infrared pixel value in the hot target region and the infrared pixel value in the cold target region; setting the infrared pixel weight curve of the background target region, the infrared pixel weight curve of the hot target region and the infrared pixel weight curve of the cold target region respectively, wherein the infrared pixel weight corresponding to the infrared pixel value in the background target region is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the hot target region; the infrared pixel weight corresponding to the infrared pixel value in the background target region is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the cold target region.

[0016] Optionally, in the step of setting the background target region, the hot target region and the cold target region in the infrared base layer pixel value range, the average value of the infrared base layer pixel value is in the background target region.

[0017] Optionally, the initial infrared pixel weight corresponding to the minimum infrared pixel value, the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value are set respectively, wherein the minimum infrared pixel value, the first infrared pixel value, the second infrared pixel value and the maximum infrared pixel value increase in turn, the initial infrared pixel weight corresponding to the minimum infrared pixel value is greater than or equal to the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the maximum infrared pixel value is greater than or equal to the initial infrared pixel weight corresponding to the second infrared pixel value, and the minimum infrared pixel value and the first infrared pixel value are in the cold target region, the first infrared pixel value and the second infrared pixel value are in the background target region, and the second infrared pixel value and the maximum infrared pixel value are in the hot target region; the infrared pixel weight curve of the cold target region is obtained according to the initial infrared pixel weight corresponding to the minimum infrared pixel value and the initial infrared pixel weight corresponding to the first infrared pixel value; the infrared pixel weight curve of the background target region is obtained according to the initial infrared pixel weight corresponding to the first infrared pixel value and the initial infrared pixel weight corresponding to the second infrared pixel value; the infrared pixel weight curve of the hot target region is obtained according to the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value.

[0018] Optionally, the step of obtaining the base layer fusion image comprises: calculating the base layer fusion image Gnr = w_vis * Gnv + w_ir * Gni according to the infrared pixel weight curve, the visible light pixel weight curve, each visible light pixel value in the visible light base layer and each infrared pixel value in the infrared base layer, wherein Gnr represents the base layer fusion image, w_vis represents the visible light pixel weight, w_ir represents the infrared pixel weight, Gnv represents the visible light pixel value, Gni represents the infrared pixel value, and n is a natural number.

[0019] Optionally, the image processing method further comprises: performing image restoration according to the 0th detail fusion image, the ath detail fusion image, the N-1th detail fusion image, and obtaining a gray-scale fusion image.

[0020] Optionally, the N-layer pyramid decomposition of the visible light original image further obtains a visible light base layer, and the N-layer pyramid decomposition of the infrared original image further obtains an infrared base layer; the image processing method further comprises: in the step of obtaining the gray-scale fusion image, performing image restoration according to the base layer fusion image, the 0th detail fusion image, the ath detail fusion image, and the N-1th detail fusion image, and obtaining the gray-scale fusion image.

[0021] Optionally, in at least one of the step of performing N-layer pyramid decomposition on the visible light original image and the step of performing N-layer pyramid decomposition on the infrared original image, N-layer Laplacian pyramid decomposition is performed on the to-be-processed image; wherein the to-be-processed image is one of the visible light original image and the infrared original image.

[0022] Optionally, the step of performing N-layer Laplacian pyramid decomposition comprises: performing the Laplacian pyramid decomposition step at least once, and performing the Laplacian pyramid decomposition step for the jth time to obtain a j-1th Laplacian layer, wherein j is a positive integer greater than or equal to 0 and less than or equal to N; wherein performing the Laplacian pyramid decomposition step for the mth time comprises: performing Gaussian filtering on an m-1th layer image; performing 1 / 2 down-sampling on the m-1th layer image subjected to the Gaussian filtering to obtain a sampled and filtered mth layer image; performing up-sampling on the mth layer image to obtain an m-1th Gaussian layer; and obtaining an m-1th Laplacian layer according to the m-1th Gaussian layer and the m-1th layer image, wherein m is a positive integer greater than or equal to 2 and less than or equal to N.

[0023] Optionally, in the step of performing the Laplacian pyramid decomposition for the first time, Gaussian filtering is performed on the to-be-processed image to obtain a 0th Laplacian layer.

[0024] Optionally, the image processing method further comprises: performing N-layer pyramid decomposition on the visible light original image to obtain a visible light base layer; performing N-layer pyramid decomposition on the infrared original image to obtain an infrared base layer, wherein N is a natural number; performing image fusion on the visible light base layer and the infrared base layer to obtain a base layer fusion image; performing image restoration according to the base layer fusion image to obtain a gray-scale fusion image; obtaining two chroma layers according to the m-1th visible light Gaussian layer and the m-1th infrared light Gaussian layer; performing m-1 times of up-sampling on the two chroma layers respectively to obtain two chroma channel images; and obtaining a color image according to the two chroma channel images and the gray-scale fusion image.

[0025] Optionally, in the step of obtaining two chroma layers according to the m-1th visible light Gaussian layer and the m-1th infrared light Gaussian layer, two chroma layers are obtained according to the N-1th visible light Gaussian layer and the N-1th infrared light Gaussian layer.

[0026] Correspondingly, the present application further provides an image processing device, comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and when the processor executes the program, the processor executes the steps of the image processing method described above.

[0027] Correspondingly, the present application further provides a computer readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, and stores computer instructions, wherein the computer instructions run to execute the steps of the image processing method described above.

[0028] Compared with the prior art, the technical scheme of the embodiment of the present application has the following beneficial effects:

[0029] The technical scheme of the present application obtains visible light detail layers and infrared detail layers by performing N-layer image decomposition on the visible light original image and the infrared original image, and performs image fusion on the ath visible light detail layer and the ath infrared detail layer according to the ath detail fusion weight. Since the visible light detail layers and the infrared detail layers include noise information, and the calculation of the detail fusion weight can adjust the noise in the visible light detail layers and the infrared detail layers, the image fusion using the detail fusion weight can reduce the noise in the visible light detail layers and the infrared detail layers, thereby ensuring the fusion effect of the visible light detail layers and the infrared detail layers.

[0030] Further, the technical scheme of the present application limits the pixel values in the infrared detail layers and the visible light detail layers by performing threshold limiting processing on the infrared pixel values in the infrared detail layers and the visible light pixel values in the visible light detail layers, so as to reduce resource occupation and inhibit the generation of the halo effect.

[0031] Further, the visible light original image is subjected to N-layer pyramid decomposition to obtain a visible light base layer; the infrared original image is subjected to N-layer pyramid decomposition to obtain an infrared base layer, and the visible light base layer and the infrared base layer are fused according to the infrared pixel weight curve and the visible light pixel weight curve, so that the display weight of different pixel values of the infrared base layer in the base layer fused image is high and low, thereby achieving the effect of 'injecting' the infrared base layer into the visible light base layer, and effectively improving the highlight degree of the infrared target after fusion; and since the visible light base layer and the infrared base layer only include structural information, the highlight display of the infrared base layer in the base layer fused image can effectively avoid introducing noise information, and can effectively guarantee the fusion effect of the visible light base layer and the infrared base layer.

[0032] Further, the background target field, the hot target field and the cold target field are set in the infrared base layer pixel value range, and the infrared pixel weight corresponding to the infrared pixel value in different target fields is different, so that the display weight of the infrared pixel value in different target fields in the base layer fused image is high and low, thereby guaranteeing that the infrared base layer can be highlighted in the base layer fused image; and the infrared pixel weight corresponding to the infrared pixel value in the cold target field and the infrared pixel weight corresponding to the infrared pixel value in the hot target field are both greater than the infrared pixel weight corresponding to the infrared pixel value in the background target field, which can effectively improve the contrast and naturalness of the fused image, and can effectively improve the fusion effect.

[0033] Further, the present application sets the initial weight value corresponding to the minimum infrared pixel value, the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value, that is, sets different initial infrared pixel weights, and obtains the infrared pixel weight curve in different target fields according to the different initial infrared pixel weights, and then the infrared pixel weight corresponding to different infrared pixel values is adjusted to obtain different infrared weight curves, so as to realize the optimization of the infrared weight curve, and make the infrared base layer accurately highlighted in the base layer fused image.

[0034] Further, according to the visible Gaussian layer and the infrared Gaussian layer obtained after N-layer pyramid decomposition, the present application obtains two chroma channel images, and performs color component processing on the two chroma channel images and the gray fused image to obtain a color output image, thereby expanding the application scenarios of image fusion. BRIEF DESCRIPTION OF DRAWINGS

[0035] Fig. 1 is a flowchart of an embodiment of the image processing method of the present application;

[0036] Figure 2 is a flow chart illustrating the steps of performing N-layer pyramid decomposition in an embodiment of the image processing method of the present application;

[0037] Figure 3 is a diagram illustrating the algorithm of performing N-layer pyramid decomposition in an embodiment of the image processing method of the present application;

[0038] Figure 4 is a diagram illustrating the simulation of performing N-layer pyramid decomposition in an embodiment of the image processing method of the present application;

[0039] Figure 5 is a flow chart illustrating the steps of performing detail layer fusion in an embodiment of the image processing method of the present application;

[0040] Figure 6 is a flow chart illustrating the steps of obtaining detail fusion weight in an embodiment of the image processing method of the present application;

[0041] Figure 7 is a diagram illustrating a curve of infrared pixel weight in an embodiment of the image processing method of the present application;

[0042] Figure 8 is a diagram illustrating the process of image restoration in an embodiment of the image processing method of the present application;

[0043] Figure 9 is a diagram illustrating the effect comparison of fusion images obtained by using different fusion methods in an embodiment of the image processing method of the present application;

[0044] Figure 10 is a diagram illustrating the effect comparison of fusion images obtained by using different fusion methods for base layer in an embodiment of the image processing method of the present application;

[0045] Figure 11 is a flow chart illustrating the steps of obtaining colorization processing in an embodiment of the image processing method of the present application;

[0046] Figure 12 is a diagram illustrating the simulation of colorization processing in an embodiment of the image processing method of the present application;

[0047] Figure 13 is a flow chart illustrating the steps of pinch operation in an embodiment of the image processing method of the present application;

[0048] Figure 14 is a diagram illustrating the simulation of pinch operation processing in an embodiment of the image processing method of the present application;

[0049] Figure 15 is a diagram illustrating another curve of infrared pixel weight in an embodiment of the image processing method of the present application;

[0050] Figure 16 is a diagram illustrating the simulation of image fusion in an embodiment of the image processing method of the present application. DETAILED DESCRIPTION

[0051] The current image fusion algorithm has the following shortcomings: 1. When the image fusion is carried out, the noise of the image is also fused, resulting in that the fused image has large noise; 2. The infrared thermal target after fusion is not prominent enough; 3. The contrast of the fused image is not as good as the original visible light image, and the natural sense of the fused image is not as good as the original visible light image; 4. The color of the fused image lacks natural sense after colorization.

[0052] To solve the above technical problems, the present application provides an image processing method, which performs N-layer pyramid decomposition on a visible light original image to obtain visible light detail layers; performs N-layer pyramid decomposition on an infrared original image to obtain infrared detail layers, wherein N is greater than or equal to 1; the image processing method further comprises: performing image fusion on the ath visible light detail layer and the ath infrared detail layer according to the ath detail fusion weight to obtain the ath detail fusion image; wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1.

[0053] Performing N-layer image decomposition on the visible light original image and the infrared original image to obtain visible light detail layers and infrared detail layers, and performing image fusion on the visible light detail layers and the infrared detail layers according to the detail fusion weight, since the visible light detail layers and the infrared detail layers include noise information, image fusion using the fusion weight can reduce the noise in the visible light detail layers and the infrared detail layers, and ensure the fusion effect of the visible light detail layers and the infrared detail layers.

[0054] To make the above-mentioned purposes, features and benefits of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0055] Referring to FIG. 1, a flowchart of an embodiment of the image processing method of the present application is shown.

[0056] The image processing method comprises:

[0057] Step S1: performing N-layer pyramid decomposition on a visible light original image to obtain visible light detail layers and a visible light base layer;

[0058] Step S2: performing N-layer pyramid decomposition on an infrared original image to obtain infrared detail layers and an infrared base layer, wherein N is greater than or equal to 1;

[0059] Step S3: performing image fusion on the ath visible light detail layer and the ath infrared detail layer according to the ath detail fusion weight to obtain the ath detail fusion image; wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1.

[0060] It should be noted that the visible light original image and the infrared original image are obtained by two different detectors. Specifically, the two images are substantially registered by a mechanical method, and then accurately registered by image scaling, stretching, cropping and the like.

[0061] First, at least one of step S1 and step S2 is performed to perform N-layer pyramid decomposition to obtain a base layer. Specifically, step S1 and step S2 can be performed in sequence; step S2 and step S1 can be performed in sequence; or step S1 and step S2 can be performed simultaneously.

[0062] In some embodiments of the present application, in the step of performing N-layer pyramid decomposition, at least one of the step of performing N-layer Laplacian pyramid decomposition on the visible light original image (step S1) and the step of performing N-layer pyramid decomposition on the infrared original image (step S2) is performed, wherein the image to be processed is one of the visible light original image and the infrared original image.

[0063] Specifically, in the step of performing N-layer pyramid decomposition on the visible light original image (step S1), N-layer Laplacian pyramid decomposition is performed on the visible light original image; and in the step of performing N-layer pyramid decomposition on the infrared original image (step S2), N-layer Laplacian pyramid decomposition is performed on the infrared original image.

[0064] In some embodiments of the example, the step of performing N-layer Laplacian pyramid decomposition comprises: performing a Laplacian pyramid decomposition step at least once, performing a j-th Laplacian pyramid decomposition step, obtaining a j-1-th Laplacian layer, and j is a positive integer greater than or equal to 0 and less than or equal to N-1.

[0065] The technical solution of performing N-layer Laplacian pyramid decomposition in an embodiment of the image processing method of the present application will be described in detail below with reference to FIG. 2, FIG. 3 and FIG. 4.

[0066] Referring to FIG. 2, a flowchart of performing a m-th Laplacian pyramid decomposition step in an embodiment of the image processing method of the present application is shown.

[0067] Specifically, the m-th Laplacian pyramid decomposition step comprises: step S11: performing Gaussian filtering on a m-1-th layer image; step S12: performing 1 / 2 downsampling on the m-1-th layer image filtered by the Gaussian filter to obtain a sampled and filtered m-th layer image; step S13: performing upsampling on the m-th layer image to obtain a m-1-th Gaussian layer; and step S14: obtaining a m-1-th Laplacian layer according to the m-1-th Gaussian layer and the m-1-th layer image, wherein m is a positive integer greater than or equal to 2 and less than or equal to N-1.

[0068] In some embodiments, in the step S11 of performing Gaussian filtering on the (m-1)th layer image, the filter kernel of the Gaussian filtering is a 3x3 filter kernel:

[0069] Specifically, the operation of the downsampling and the Gaussian filtering before the downsampling is that, first, edge padding and Gaussian filtering are performed, and then 1 / 2 downsampling is performed, wherein the edge padding adopts a mode of copying edges to fill one row and one column at each of the four edges of the image; the filter kernel of the Gaussian filtering is the same as above, and then when performing 1 / 2 downsampling, the values of even rows and even columns are reserved as the downsampling result (counting from 0 row and 0 column, 0 row is counted as an even row, and 0 column is counted as an even column).

[0070] Specifically, the operation method of the upsampling is that, one row of 0 is inserted after each row of the image, and one row of 0 is inserted after each column of the image, to obtain an image with double width and height, and it is noted that odd rows and odd columns are 0 (counting from 0 row and 0 column, 0 row is counted as an even row, and 0 column is counted as an even column).

[0071] In other embodiments of the present application, in the step of performing Gaussian filtering on the (m-1)th layer image when resources are sufficient, the filter kernel of the Gaussian filtering can also be a 5x5 filter kernel:

[0072] In the above scheme, the fusion image obtained by using the 5x5 filter kernel has a more natural edge transition.

[0073] In combination with reference to FIG. 3 and FIG. 4, wherein FIG. 3 shows an algorithm schematic diagram of performing N-layer pyramid decomposition in the image processing method of the present application; and FIG. 4 shows a simulation schematic diagram of performing N-layer pyramid decomposition in the image processing method of the present application.

[0074] As shown in FIG. 3, in the first time of performing the Laplacian pyramid decomposition step, Gaussian filtering is performed on the 0th layer image G0 1 / 2 downsampling is performed on the Gaussian filtered 0th layer image to obtain a sampled and filtered 1st layer image G1; upsampling is performed on the sampled and filtered 1st layer image G1 to obtain a 0th Gaussian layer G0e; and the 0th Laplacian layer L0 is obtained according to the 0th Gaussian layer G0e and the 0th layer image G0.

[0075] That is, L0=G0-G0e.

[0076] It should be noted that in some embodiments, the step of performing the Laplacian pyramid decomposition for the first time includes performing Gaussian filtering on a to-be-processed image to obtain a 0th Laplacian layer, wherein the to-be-processed image is one of the visible light original image and the infrared original image.

[0077] Specifically, in the process of N-layer pyramid decomposition of the visible light original image, the first time of performing the Laplacian pyramid decomposition step includes: performing Gaussian filtering on the visible light original image (as shown in the image G0 on the left side of FIG. 4) performing 1 / 2 down-sampling on the Gaussian-filtered 0th layer image to obtain a sampled and filtered 1st layer image G1 (as shown in the image G1 on the left side of FIG. 4); performing up-sampling on the sampled and filtered 1st layer image G1 to obtain a 0th Gaussian layer G0e (as shown in the image G0e on the left side of FIG. 4); and obtaining a 0th Laplacian layer L0 (as shown in the image L0 on the left side of FIG. 4) according to the 0th Gaussian layer G0e and the 0th layer image G0.

[0078] Specifically, in the process of N-layer pyramid decomposition of the infrared original image, the first time of performing the Laplacian pyramid decomposition step includes: performing Gaussian filtering on the infrared original image (as shown in the image G0 on the right side of FIG. 4) performing 1 / 2 down-sampling on the Gaussian-filtered 0th layer image to obtain a sampled and filtered 1st layer image G1 (as shown in the image G1 on the right side of FIG. 4); performing up-sampling on the sampled and filtered 1st layer image G1 to obtain a 0th Gaussian layer G0e (as shown in the image G0e on the right side of FIG. 4); and obtaining a 0th Laplacian layer L0 (as shown in the image L0 on the right side of FIG. 4) according to the 0th Gaussian layer G0e and the 0th layer image G0.

[0079] With reference back to FIG. 3, in the second time of performing the Laplacian pyramid decomposition step,

[0080] performing Gaussian filtering on the 1st layer image G1 performing 1 / 2 down-sampling on the Gaussian-filtered 1st layer image to obtain a sampled and filtered 2nd layer image G2; performing up-sampling on the sampled and filtered 2nd layer image G2 to obtain a 1st Gaussian layer G1e; and obtaining a 1st Laplacian layer L1 according to the 1st Gaussian layer G1e and the 1st layer image G1.

[0081] That is, L1=G1-G1e.

[0082] Specifically, in the process of N-layer pyramid decomposition of the visible light original image, the second time of performing the Laplacian pyramid decomposition step includes: performing Gaussian filtering on the 1st layer image G1 (as shown in the image G1 on the left side of FIG. 4) obtained based on the visible light original image down-sampling the 1st layer image G1 filtered by the Gaussian filter by 1 / 2 to obtain a 2nd layer image G2 filtered by the sampling (as shown in the image G2 on the left side of Fig. 4); up-sampling the 2nd layer image G2 filtered by the sampling to obtain a 1st Gaussian layer G1e (as shown in the image G1e on the left side of Fig. 4); and obtaining a 1st Laplacian layer L1 according to the 1st Gaussian layer G1e and the 1st layer image G1 (as shown in the image L1 on the left side of Fig. 4).

[0083] Specifically, in the process of performing the N-layer pyramid decomposition on the infrared original image, the 2nd time of performing the Laplacian pyramid decomposition step includes: performing a Gaussian filter on the 1st layer image G1 (as shown in the image G1 on the right side of Fig. 4) obtained based on the infrared original image down-sampling the 1st layer image G1 filtered by the Gaussian filter by 1 / 2 to obtain a 2nd layer image G2 filtered by the sampling (as shown in the image G2 on the right side of Fig. 4); up-sampling the 2nd layer image G2 filtered by the sampling to obtain a 1st Gaussian layer G1e (as shown in the image G1e on the right side of Fig. 4); and obtaining a 1st Laplacian layer L1 according to the 1st Gaussian layer G1e and the 1st layer image G1 (as shown in the image L1 on the right side of Fig. 4).

[0084] With reference to Fig. 3, in the 3rd time of performing the Laplacian pyramid decomposition step, a Gaussian filter is performed on the 2nd layer image G2 down-sampling the 2nd layer image G2 filtered by the Gaussian filter by 1 / 2 to obtain a 3rd layer image G3 filtered by the sampling; up-sampling the 3rd layer image G3 filtered by the sampling to obtain a 2nd Gaussian layer G2e; and obtaining a 2nd Laplacian layer L2 according to the 2nd Gaussian layer G2e and the 2nd layer image G2.

[0085] That is, L1=G2-G2e.

[0086] Specifically, in the process of performing the N-layer pyramid decomposition on the visible light original image, the 3rd time of performing the Laplacian pyramid decomposition step includes: performing a Gaussian filter on the 2nd layer image G2 (as shown in the image G2 on the left side of Fig. 4) obtained based on the visible light original image down-sampling the 2nd layer image G2 filtered by the Gaussian filter by 1 / 2 to obtain a 3rd layer image G3 filtered by the sampling (as shown in the image G3 on the left side of Fig. 4); up-sampling the 3rd layer image G3 filtered by the sampling to obtain a 2nd Gaussian layer G2e (as shown in the image G1e on the left side of Fig. 4); and obtaining a 2nd Laplacian layer L2 according to the 2nd Gaussian layer G2e and the 2nd layer image G2 (as shown in the image L2 on the left side of Fig. 4).

[0087] Specifically, in the step S2, the third time of performing the Laplacian pyramid decomposition step in the process of performing the N-layer pyramid decomposition on the infrared original image comprises: performing a Gaussian filtering on the second layer image G2 (as shown in the image G2 on the right side of the Fig. 4) obtained based on the infrared original image performing a 1 / 2 down-sampling on the second layer image G2 after the Gaussian filtering to obtain a sampled and filtered third layer image G3 (as shown in the image G3 on the right side of the Fig. 4); performing an up-sampling on the sampled and filtered third layer image G3 to obtain a second Gaussian layer G2e (as shown in the image G1e on the right side of the Fig. 4); and obtaining a second Laplacian layer L2 (as shown in the image L2 on the right side of the Fig. 4) according to the second Gaussian layer G2e and the second layer image G2.

[0088] The mth time of performing the Laplacian pyramid decomposition step is as described in the above solution, and thus the present application is not described here again by analogy.

[0089] In some embodiments of the example, in the step of performing the N-layer pyramid decomposition, a three-layer Laplacian pyramid decomposition is performed; and the step of performing the three-layer Laplacian pyramid decomposition comprises: performing the Laplacian pyramid decomposition step for three times.

[0090] After performing the three times of the Laplacian pyramid decomposition step, four layers of Gaussian pyramid and three layers of Laplacian pyramid are obtained, wherein the four layers of Gaussian pyramid comprise: the 0th layer image G0, the 1st layer image G1, the 2nd layer image G2 and the 3rd layer image G3, and the three layers of Laplacian pyramid comprise: the 0th Laplacian layer L0, the 1st Laplacian layer L1 and the 2nd Laplacian layer L2.

[0091] As shown in the Fig. 4, the 0th layer image G0 on the left side, the 1st layer image G1 on the left side, the 2nd layer image G2 on the left side and the 3rd layer image G3 on the left side are respectively the visible light original image and the corresponding sampled and filtered 1st layer image G1, the sampled and filtered 2nd layer image G2 and the sampled and filtered 3rd layer image G3 in the four layers of Gaussian pyramid obtained by performing the three-layer Laplacian pyramid decomposition on the visible light original image; the 0th Laplacian layer L0 on the left side, the 1st Laplacian layer L1 on the left side and the 2nd Laplacian layer L2 on the left side are respectively the 0th Laplacian layer L0, the 1st Laplacian layer L1 and the 2nd Laplacian layer L2 in the three layers of Laplacian pyramid obtained by performing the three-layer Laplacian pyramid decomposition on the visible light original image.

[0092] As shown in FIG. 4, the right 0th layer image G0, the right 1st layer image G1, the right 2nd layer image G2 and the right 3rd layer image G3 are respectively the infrared original image and the corresponding sampled filtered 1st layer image G1, the sampled filtered 2nd layer image G2 and the sampled filtered 3rd layer image G3 in the 4-layer Gaussian pyramid obtained by performing 3-layer Laplacian pyramid decomposition on the infrared original image; the left 0th layer image L0, the left 1st layer image L1 and the left 2nd layer image L2 are respectively the 0th Laplacian layer L0, the 1st Laplacian layer L1 and the 2nd Laplacian layer L2 in the 3-layer Laplacian pyramid obtained by performing 3-layer Laplacian pyramid decomposition on the infrared original image.

[0093] In some embodiments of the present application, at least one of the step S1 and the step S2 is performed, and in the step of performing N-layer pyramid decomposition, a detail layer is also obtained.

[0094] The base layer is a thumbnail of the original image, and after N-layer pyramid decomposition, the base layer contains very low noise information, almost no noise, and the base layer only carries the structural basic information of the original image.

[0095] The detail layer only carries texture, edge, noise and other information, and therefore the fusion strategy of the base layer is the most critical, which relates to the basic framework, contrast and highlight degree of the infrared target of the fusion image.

[0096] Specifically, with reference back to FIG. 1, in the step of performing N-layer pyramid decomposition, the base layer is obtained, that is, in the step S1, the N-layer pyramid decomposition is performed on the visible light original image to obtain a visible light base layer; and in the step S2, the N-layer pyramid decomposition is performed on the infrared original image to obtain an infrared base layer.

[0097] Specifically, with reference back to FIG. 1, in the step S1 of performing N-layer pyramid decomposition on the visible light original image, N-layer visible light detail layers are also obtained; and in the step S2 of performing N-layer pyramid decomposition on the infrared original image, N-layer infrared detail layers are also obtained.

[0098] In some embodiments of the present application, the highest layer of the Gaussian pyramid, that is, the sampled filtered Nth layer image, is the base layer; and each Laplacian layer in the Laplacian pyramid is a detail layer.

[0099] Specifically, the highest layer of the Gaussian pyramid obtained by performing N-layer Laplacian pyramid decomposition on the visible light original image is the visible light base layer; and the highest layer of the Gaussian pyramid obtained by performing N-layer Laplacian pyramid decomposition on the infrared original image is the infrared base layer.

[0100] Specifically, the N-layer Laplacian pyramid obtained by performing N-layer Laplacian pyramid decomposition on the visible light original image is N-layer visible light detail layers; the N-layer Laplacian pyramid obtained by performing N-layer Laplacian pyramid decomposition on the infrared original image is N-layer infrared detail layers.

[0101] Specifically, the 0th Laplacian layer L0, the 1st Laplacian layer L1 and the 2nd Laplacian layer L2 in the N-layer Laplacian pyramid obtained by performing N-layer Laplacian pyramid decomposition on the visible light original image are respectively the 0th visible light detail layer, the 1st visible light detail layer and the 2nd visible light detail layer; the 0th Laplacian layer L0, the 1st Laplacian layer L1 and the 2nd Laplacian layer L2 in the N-layer Laplacian pyramid obtained by performing N-layer Laplacian pyramid decomposition on the infrared original image are respectively the 0th infrared detail layer, the 1st infrared detail layer and the 2nd infrared detail layer.

[0102] With reference to FIG. 1, in some embodiments of the present application, the N-layer pyramid decomposition on the visible light original image in step S1 further obtains N-layer visible light detail layers, wherein the N-layer visible light detail layers include: the 0th visible light detail layer, the 1st visible light detail layer, …, the N-1th visible light detail layer; the N-layer pyramid decomposition on the infrared original image in step S2 further obtains N-layer infrared detail layers, wherein the N-layer infrared detail layers include: the 0th infrared detail layer, the 1st infrared detail layer, …, the N-1th infrared detail layer.

[0103] In some embodiments as shown in FIG. 4, N=3; the 3-layer visible light detail layers include: the 0th visible light detail layer, the 1st visible light detail layer and the 2nd visible light detail layer, which are respectively the 0th Laplacian layer L0, the 1st Laplacian layer L1 and the 2nd Laplacian layer L2 in the 3-layer Laplacian pyramid obtained by performing 3-layer Laplacian pyramid decomposition on the visible light original image; the 3-layer infrared detail layers include: the 0th infrared detail layer, the 1st infrared detail layer and the 2nd infrared detail layer, which are respectively the 0th Laplacian layer L0, the 1st Laplacian layer L1 and the 2nd Laplacian layer L2 in the 3-layer Laplacian pyramid obtained by performing 3-layer Laplacian pyramid decomposition on the infrared original image.

[0104] As shown in FIG. 1, in some embodiments of the present application, the image processing method further includes: step S3, performing image fusion on the ath visible light detail layer and the ath infrared detail layer to obtain an ath detail fusion image; wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1.

[0105] In some embodiments of the present application, N is greater than or equal to 1 and less than or equal to 5.

[0106] Specifically, with continuous reference to FIG. 4, when the visible light original image is decomposed into a 3-layer Laplacian pyramid, the 0th Laplacian layer L0, the 1st Laplacian layer L1 and the 2nd Laplacian layer L2 are respectively the 0th visible light detail layer, the 1st visible light detail layer and the 2nd visible light detail layer; when the infrared original image is decomposed into a 3-layer Laplacian pyramid, the 0th Laplacian layer L0, the 1st Laplacian layer L1 and the 2nd Laplacian layer L2 are respectively the 0th infrared detail layer, the 1st infrared detail layer and the 2nd infrared detail layer.

[0107] Specifically, as shown in some embodiments of FIG. 4, the image processing method further comprises: performing image fusion on the 0th visible light detail layer and the 0th infrared detail layer to obtain a 0th detail fusion image F0 (as shown in image F0 in FIG. 4); performing image fusion on the 1st visible light detail layer and the 1st infrared detail layer to obtain a 1st detail fusion image F1 (as shown in image F1 in FIG. 4); and performing image fusion on the 2nd visible light detail layer and the 2nd infrared detail layer to obtain a 2nd detail fusion image F2 (as shown in image F2 in FIG. 4).

[0108] As shown in FIG. 5, in some embodiments of the present application, the step S3 of performing image fusion on the a th visible light detail layer and the a th infrared detail layer to obtain an a th detail fusion image comprises: a step S32 of performing image fusion on the a th visible light detail layer and the a th infrared detail layer according to an a th detail fusion weight to obtain an a th detail fusion image.

[0109] Specifically, as shown in FIG. 5, in some embodiments, the step S3 of performing image fusion on the a th visible light detail layer and the a th infrared detail layer to obtain an a th detail fusion image further comprises: a step S31 of obtaining an a th detail fusion weight.

[0110] Referring to FIG. 6, FIG. 6 shows a flowchart of the step of obtaining a detail fusion weight in the image processing method of the present application.

[0111] In some embodiments of the present application, the step S31 of obtaining an a th detail fusion weight comprises:

[0112] A step S311 of obtaining an activity of the a th infrared detail layer according to the a th infrared detail layer;

[0113] A step S312 of obtaining an activity of the a th visible light detail layer according to the a th visible light detail layer;

[0114] A step S313 of obtaining an a th detail fusion weight according to the activity of the a th infrared detail layer and the activity of the a th visible light detail layer, and combining an a th infrared noise reduction parameter and an a th visible light noise reduction parameter.

[0115] For example, the filter kernel of the mean filtering is a 1x5 filter kernel: In other embodiments, a larger filter kernel can also be selected when resources are sufficient.

[0116] In some embodiments, the step of obtaining the activity of the detail layer includes mean filtering the absolute value of the detail layer to obtain the activity of the detail layer. Specifically, in the step of obtaining the activity of the a-th infrared detail layer according to the a-th infrared detail layer, the absolute value of the a-th infrared detail layer is mean filtered to obtain the activity act_ir of the a-th infrared detail layer: act_ir = blur(abs(L2i)); and in the step of obtaining the activity of the a-th visible light detail layer according to the a-th visible light detail layer, the absolute value of the a-th visible light detail layer is mean filtered to obtain the activity act_vi of the a-th visible light detail layer: act_vi = blur(abs(L2v)).

[0117] In the step of obtaining the a-th layer detail fusion weight, the a-th infrared noise reduction parameter and the a-th visible light noise reduction parameter are suitable parameters for adjusting the respective weights of the a-th infrared detail layer and the a-th visible light detail layer to suppress noise. In some embodiments, when the original image has a large noise, adjusting the a-th infrared noise reduction parameter and the a-th visible light noise reduction parameter can help reduce noise.

[0118] As shown in FIG. 4, after obtaining the a-th detail fusion weight, the a-th visible light detail layer and the a-th infrared detail layer are image fused according to the a-th detail fusion weight to obtain an a-th detail fusion image.

[0119] Specifically, after the step of obtaining the a-th detail fusion weight, the a-th detail layer fusion image Lar = Lav*weight + Lai*(1-weight) is calculated according to the detail fusion weight, each infrared pixel value in the infrared detail layer, and each visible light pixel value in the visible light detail layer. Lavi Lavi Lavi Lar represents the a-th detail layer fusion image, weight represents the a-th detail fusion weight, Lav is the a-th visible light detail layer, and Lai is the a-th infrared detail layer.

[0120] In some embodiments as shown in FIG. 4, the second visible light detail layer is the second Laplacian layer L2 in the Laplacian pyramid obtained by performing 3-layer Laplacian pyramid decomposition on the visible light original image; and the second infrared detail layer is the second Laplacian layer L2 in the Laplacian pyramid obtained by performing 3-layer Laplacian pyramid decomposition on the infrared original image.

[0121] ​​The second Laplacian layer L2 contains large edges in the image with relatively less noise. A weight can be calculated according to the activity, and then the images are fused by weighted average (WA) according to the weight.

[0122] According to the a-th infrared detail layer, the activity of the a-th infrared detail layer is obtained by the following steps: taking the absolute value of the second infrared detail layer L2i to obtain the absolute value of the second infrared detail layer abs(L2i); and performing mean filtering on the absolute value of the second infrared detail layer abs(L2i) using a 1x5 filter kernel to obtain the activity act_ir of the second infrared detail layer L2i: act_ir = blur(abs(L2i)), where blur is a filter function and abs is an absolute value function.

[0123] According to the a-th visible light detail layer, the activity of the a-th visible light detail layer is obtained by the following steps: taking the absolute value of the second visible light detail layer L2v to obtain the absolute value of the second visible light detail layer abs(L2v); and performing mean filtering on the absolute value of the second visible light detail layer abs(L2v) using a 1x5 filter kernel to obtain the activity act_v of the second visible light detail layer L2v: act_v = blur(abs(L2v)), where blur is a filter function and abs is an absolute value function.

[0124] It should be noted that the 1x5 mean filtering filter kernel is: Using a filter kernel in the form of a row can save the design of a line buffer. In the case of sufficient resources, a larger filter kernel can also be selected.

[0125] Then, the second detail fusion weight is calculated:

[0126] wherein weight L2vi represents the second detail fusion weight, k L2vi and k L2ir respectively represent the second visible light denoising parameter and the second infrared denoising parameter, and take integer values between 1 and 64, with an initial value of 1. When the noise of one of the second visible light detail layer and the second infrared detail layer is large, adjusting the second visible light denoising parameter k L2vi and the second infrared denoising parameter k L2ir can help reduce noise. Then, the second detail layer fusion image L2r = L2v*weight L2vi + L2i*(1-weight L2vi ) is calculated according to the detail fusion weight, the infrared pixel values in the infrared detail layer, and the visible light pixel values in the visible light detail layer, where L2r represents the second detail layer fusion image, weight L2virepresents the 2nd detail fusion weight, L2v is the 2nd visible light detail layer, and L2i is the 2nd infrared detail layer.

[0127] Then, the 1st detail fusion weight is calculated as follows:

[0128] weight L1vi represents the 1st detail fusion weight, k L1vi and k L1ir are respectively the 1st visible light noise reduction parameter and the 1st infrared noise reduction parameter, and take integer values between 1 and 64, with an initial value of 1. When the noise of one of the 1st visible light detail layer and the 1st infrared detail layer is large, adjusting the 1st visible light noise reduction parameter k L1vi and the 1st infrared noise reduction parameter k L1ir can help reduce noise. Then, according to the detail fusion weight, each infrared pixel value in the infrared detail layer, and each visible light pixel value in the visible light detail layer, the 1st detail layer fusion image L1r = L1v * weight L1vi + L1i * (1 - weight L1vi ) is calculated, where L1r represents the 1st detail layer fusion image, weight L1vi represents the 1st detail fusion weight, L1v is the 1st visible light detail layer, and L1i is the 1st infrared detail layer.

[0129] Then, the 0th detail fusion weight is calculated as follows:

[0130] weight L0vi represents the 0th detail fusion weight, k L0vi and k L0ir are respectively the 0th visible light noise reduction parameter and the 0th infrared noise reduction parameter, and take integer values between 1 and 64, with an initial value of 1. When the noise of one of the 0th visible light detail layer and the 0th infrared detail layer is large, adjusting the 0th visible light noise reduction parameter k L0vi and the 0th infrared noise reduction parameter k L0ir can help reduce noise. Then, according to the detail fusion weight, each infrared pixel value in the infrared detail layer, and each visible light pixel value in the visible light detail layer, the 0th detail layer fusion image L0r = L0v * weight L0vi + L0i * (1 - weight L0vi ) is calculated, where L0r represents the 0th detail layer fusion image, weight L0vi represents the 0th detail fusion weight, L0v is the 0th visible light detail layer, and L0i is the 0th infrared detail layer.

[0131] In the above scheme, the visible light original image is subjected to N-layer pyramid decomposition to obtain a visible light detail layer, and the infrared original image is subjected to N-layer pyramid decomposition to obtain an infrared detail layer, and the visible light detail layer and the infrared detail layer are fused. Since the visible light detail layer and the infrared detail layer include noise information, different weight parameters are set, and the fusion weight of the infrared detail layer and the fusion weight of the visible light detail layer are obtained according to the weight parameters and the activity, which can effectively suppress the influence of noise in the visible light detail layer and the infrared detail layer on the fusion effect, and can ensure the fusion effect of the visible light detail layer and the infrared detail layer.

[0132] With continuous reference to FIG. 1, after obtaining the visible light base layer and the infrared light base layer, the image processing method further comprises: step S4, fusing the visible light base layer and the infrared base layer according to the infrared pixel weight curve and the visible light pixel weight curve to obtain a base layer fusion image.

[0133] The infrared pixel weight relationship is suitable for characterizing the proportion of the infrared pixel value in the obtained base layer fusion image. Based on fusing the visible light base layer and the infrared base layer according to the infrared pixel weight curve and the visible light pixel weight curve, the display weight of different pixel values of the infrared base layer in the base layer fusion image is high or low, so as to achieve the effect of “injecting” the infrared base layer into the visible light base layer, which can effectively improve the prominence of the infrared target after fusion; and since only structural information is included in the visible light base layer and the infrared base layer, the prominent display of the infrared base layer in the base layer fusion image can effectively avoid introducing noise information, and can effectively ensure the fusion effect of the visible light base layer and the infrared base layer.

[0134] In some embodiments of the present application, as shown in FIG. 1, the image processing method further comprises: setting an infrared pixel weight curve.

[0135] Specifically, with continuous reference to FIG. 1, the step of setting the infrared pixel weight curve comprises:

[0136] Step S5: setting a background target region, a hot target region and a cold target region in the infrared base layer pixel value range, wherein the infrared pixel value in the background target region is between the infrared pixel value in the hot target region and the infrared pixel value in the cold target region;

[0137] Step S6: respectively setting an infrared pixel weight curve of the background target region, an infrared pixel weight curve of the hot target region and an infrared pixel weight curve of the cold target region;

[0138] The infrared pixel weight corresponding to the infrared pixel value in the background target field is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the hot target field; and the infrared pixel weight corresponding to the infrared pixel value in the background target field is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the cold target field.

[0139] In some embodiments of the present application, in the step S5 of setting the background target field, the hot target field and the cold target field within the range of the infrared base layer pixel value, the average value Xmean of the infrared base layer pixel value is located in the background target field.

[0140] The average value Xmean of the pixel value is close to the background brightness, and the average value Xmean of the infrared base layer pixel value is close to the background brightness of the infrared base layer. In the background target field, the hot target field and the cold target field divided by locating the average value Xmean of the infrared base layer pixel value in the background target field, the background target field is located around the background brightness of the infrared base layer, and the hot target field and the cold target field are both deviated from the background brightness. In addition, the method of dividing the background target field, the hot target field and the cold target field by locating the average value Xmean of the infrared base layer pixel value in the background target field makes the set infrared pixel weight curve a dynamic curve, and the set infrared pixel weight curve will change with the change of the average value Xmean of the infrared base layer pixel value.

[0141] In some embodiments of the present application, the range of the infrared base layer pixel value is 0 to 256, the average value of the infrared base layer pixel value is real-time statistics, and when the absolute value of the difference between the current infrared base layer pixel value and the average value of the infrared base layer pixel value is greater than a preset difference threshold value, the current infrared base layer pixel value is located in the cold target field or the hot target field. For example, the preset difference threshold value is 40.

[0142] When the difference between the current infrared base layer pixel value and the average value of the infrared base layer pixel value is greater than a preset difference threshold value, and the difference is a positive number, the current infrared base layer pixel value is located in the hot target field.

[0143] When the difference between the current infrared base layer pixel value and the average value of the infrared base layer pixel value is greater than a preset difference threshold value, and the difference is a negative number, the current infrared base layer pixel value is located in the cold target field.

[0144] Referring to FIG. 7, FIG. 7 shows a schematic diagram of an infrared pixel weight curve in the image processing method of the present application. In the figure, the X-axis coordinate is represented as the infrared pixel value IR, and the Y-axis coordinate is represented as the infrared pixel weight weight.

[0145] In some embodiments of the present application, the step S5 of setting the infrared pixel weight curve of the background target region, the infrared pixel weight curve of the hot target region and the infrared pixel weight curve of the cold target region respectively comprises: setting the initial infrared pixel weight W1 corresponding to the minimum infrared pixel value Xmin, the initial infrared pixel weight W2 corresponding to the first infrared pixel value X1, the initial infrared pixel weight W3 corresponding to the second infrared pixel value X2 and the initial infrared pixel weight W4 corresponding to the maximum infrared pixel value Xmax respectively.

[0146] Wherein, the minimum infrared pixel value Xmin, the first infrared pixel value X1, the second infrared pixel value X2 and the maximum infrared pixel value Xmax increase in turn, the minimum infrared pixel value Xmin corresponding to the initial infrared pixel weight W1 is greater than or equal to the first infrared pixel value X1 corresponding to the initial infrared pixel weight W2, the maximum infrared pixel value Xmax corresponding to the initial infrared pixel weight W4 is greater than or equal to the second infrared pixel value X2 corresponding to the initial infrared pixel weight W3, and the minimum infrared pixel value Xmin and the first infrared pixel value X1 are the cold target region, the first infrared pixel value X1 and the second infrared pixel value X2 are the background target region, and the second infrared pixel value X2 and the maximum infrared pixel value Xmax are the hot target region.

[0147] In some embodiments of the present application, the step S6 of setting the infrared pixel weight curve of the background target region, the infrared pixel weight curve of the hot target region and the infrared pixel weight curve of the cold target region respectively further comprises: obtaining the infrared pixel weight curve of the cold target region according to the initial infrared pixel weight corresponding to the minimum infrared pixel value and the initial infrared pixel weight corresponding to the first infrared pixel value; obtaining the infrared pixel weight curve of the background target region according to the initial infrared pixel weight corresponding to the first infrared pixel value and the initial infrared pixel weight corresponding to the second infrared pixel value; and obtaining the infrared pixel weight curve of the hot target region according to the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value.

[0148] As shown in FIG. 7, the infrared pixel weight curve of the hot target region and the infrared pixel weight curve of the cold target region are both greater than the infrared pixel weight curve of the background target region, that is, the weight of the infrared base layer of the target significantly deviating from the background brightness in the base layer fusion image is increased to appropriately highlight: the target whose pixel value is significantly greater than the average value Xmean of the infrared base layer pixel value, that is, the "hot target" whose temperature is significantly higher than the background temperature, is a more important target in the infrared image (for example, a car driving in a forest), which needs to be highlighted, and a higher weight is set; the target whose pixel value is significantly less than the average value Xmean of the infrared base layer pixel value, that is, the "cold target" whose temperature is significantly lower than the background temperature, is also a more important target in the infrared image (for example, a vehicle that has been stationary in a forest for a period of time), which also needs to be highlighted, and a higher weight is also set.

[0149] Specifically, as shown in FIG. 7, the abscissa of the first endpoint A1 is the minimum infrared pixel value Xmin, and the ordinate of the first endpoint A1 is the initial infrared pixel weight W1 corresponding to the minimum infrared pixel value Xmin; the abscissa of the second endpoint A2 is the first infrared pixel value X1, and the ordinate of the second endpoint A2 is the initial infrared pixel weight W2 corresponding to the first infrared pixel value X1; the abscissa of the third endpoint A3 is the second infrared pixel value X2, and the ordinate of the third endpoint A3 is the initial infrared pixel weight W3 corresponding to the second infrared pixel value X2; the abscissa of the fourth endpoint A4 is the maximum infrared pixel value Xmax, and the ordinate of the fourth endpoint A4 is the initial infrared pixel weight W4 corresponding to the maximum infrared pixel value Xmax.

[0150] With reference to FIG. 7, the acquisition of the infrared pixel weight curve is specifically connecting the first endpoint A1 and the second endpoint A2 to obtain the infrared pixel weight curve L1 of the cold target region; connecting the second endpoint A2 and the third endpoint A3 to obtain the infrared pixel weight curve L2 of the background target region; and connecting the third endpoint A3 and the fourth endpoint A4 to obtain the infrared pixel weight curve L3 of the hot target region.

[0151] In some embodiments as shown in FIG. 7, the coordinates of the first endpoint A1 are (0.3, 0), the coordinates of the second endpoint A2 are (0.1, xmean-dx1), the coordinates of the third endpoint A3 are (0, xmean+dx2), and the coordinates of the fourth endpoint A4 are (0.9, 255), where dx1 is the distance between xmean and the abscissa of the second endpoint A2, and dx2 is the distance between xmean and the abscissa of the third endpoint A3.

[0152] With reference back to FIG. 4, after the infrared pixel weight curve is obtained, each infrared pixel in the infrared base layer is "injected" into the visible light base layer according to the corresponding infrared pixel weight of each infrared pixel in the infrared pixel weight curve, which can effectively improve the prominence of the infrared target after fusion and effectively improve the contrast and naturalness of the image after fusion, where the image after fusion of the infrared base layer and the visible light base layer is F3.

[0153] In the above scheme, by setting the initial weight value corresponding to the minimum infrared pixel value, the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the second infrared pixel value, and the initial infrared pixel weight corresponding to the maximum infrared pixel value, i.e., setting different initial infrared pixel weights, and obtaining the infrared pixel weight curve in different target fields according to the different initial infrared pixel weights, the subsequent different infrared pixel weight curves can be obtained by adjusting the infrared pixel weight corresponding to different infrared pixel values, which realizes the optimization of the infrared weight curve and can effectively improve the prominence of the infrared target after fusion.

[0154] With reference back to FIG. 1, in some embodiments of the present application, the image processing method further comprises: step S7: setting a visible light pixel weight curve.

[0155] In some embodiments of the present application, the sum of the visible light pixel weight and the infrared pixel weight is 1 for the same pixel value; in the step of setting the visible light pixel weight curve in step S7, the visible light pixel weight curve is set according to the infrared pixel weight curve.

[0156] In some embodiments of the example, the weight of each visible light pixel in the visible light pixel weight curve is 1-w_ir, where w_ir is each infrared pixel weight.

[0157] With reference back to FIG. 1, in some embodiments, in the step of performing image fusion on the visible light base layer and the infrared base layer to obtain a base layer fusion image in step S4, the base layer fusion image Gnr is calculated according to the infrared pixel weight curve, the visible light pixel weight curve, each visible light pixel value in the visible light base layer, and each infrared pixel value in the infrared base layer: Gnr=w_vis*Gnv+w_ir*Gni, where Gnr represents the base layer fusion image, w_vis represents the visible light pixel weight, w_ir represents the infrared pixel weight, Gnv represents the visible light pixel value, Gni represents the infrared pixel value, and n∈N, n is a natural number.

[0158] It should be noted that in some embodiments of the present application, the visible light base layer and the infrared base layer are image fused according to the infrared pixel weight curve and the visible light pixel weight curve to obtain a base layer fusion image. However, this approach is only an example.

[0159] In other embodiments of the present application, the visible light base layer and the infrared base layer are image fused to obtain a base layer fusion image, and the visible light base layer and the infrared base layer are average fused to obtain the base layer fusion image.

[0160] With reference to FIG. 1, in some embodiments of the present application, the image processing method further includes: step 8, image restoration is performed according to the base layer fusion image to obtain a gray fusion image.

[0161] In some embodiments, step S1, the visible light original image is decomposed into N layers of pyramid, and N layers of visible light detail layers are also obtained; step S2, the infrared original image is decomposed into N layers of pyramid, and N layers of infrared detail layers are also obtained; the image processing method further includes: step 3, the a th visible light detail layer and the a th infrared detail layer are image fused to obtain an a th detail fusion image; step 8, in the step of image restoration according to the base layer fusion image to obtain a gray fusion image, the base layer fusion image, the 0 th detail fusion image, …, the a th detail fusion image, …, and the N-1 th detail fusion image are used for image restoration to obtain the gray fusion image.

[0162] For example, in some embodiments as shown in FIG. 4, in the step of image restoration according to the base layer fusion image to obtain a gray fusion image, the base layer fusion image, the 2 nd detail fusion image, the 1 st detail fusion image, and the 0 th detail fusion image are used for image restoration to obtain the gray fusion image.

[0163] In some embodiments, step S8, the step of image restoration according to the base layer fusion image to obtain a gray fusion image includes: obtaining an n th restoration image GRn, 0≤n≤N; wherein the step of obtaining the n th restoration image GRn includes: up-sampling the n+1 th restoration image; and obtaining the n th restoration image GRn according to the sum of the up-sampled n+1 th restoration image and the n th detail fusion image.

[0164] In some embodiments, step S8, the step of image restoration according to the base layer fusion image to obtain a gray fusion image further includes: sequentially obtaining an n th restoration image GRn, an n+1 th restoration image GRn+1, …, and until a 0 th restoration image GR0, wherein the 0 th restoration image GR0 is the gray fusion image.

[0165] It should be noted that in step S8, the step of performing image restoration according to the base layer fusion image to obtain a gray-scale fusion image, the base layer fusion image is taken as the Nth restored image GRN.

[0166] In some embodiments as shown in FIG. 4 and FIG. 8, in step S8, the step of performing image restoration according to the base layer fusion image to obtain a gray-scale fusion image, the third layer restored image GR3, the second layer restored image GR2, the first layer restored image GR1 and the zeroth layer restored image GR0 obtained in sequence are respectively:

[0167] The base layer fusion image F3 is taken as the Nth restored image GR3; the third restored image GR3 is up-sampled to obtain a sampled third restored image GR3↑, and the second restored image GR2 is obtained according to the sampled third restored image GR3↑ and L2; the second restored image GR2 is up-sampled to obtain a sampled second restored image GR2↑, and the first restored image GR1 is obtained according to the sampled second restored image GR2↑ and L1; the first restored image GR1 is up-sampled to obtain a sampled first restored image GR1↑, and the zeroth restored image GR0 is obtained according to the sampled first restored image GR1↑ and L0.

[0168] The zeroth restored image GR0 is a gray-scale fusion image.

[0169] Referring to FIG. 9, the effect of comparing the technical scheme of the present application with other technologies is compared, and the technical effect of the present application is discussed again. FIG. 9 shows an effect comparison diagram of fusion images obtained by using different fusion methods in the image processing method of the present application. In FIG. 9, (a) is a visible light original image, (b) is an infrared original image, (c) is a fusion image obtained by using a fusion method of the prior art (for example, average fusion), and (d) is a fusion image obtained by using the fusion method of the present application.

[0170] As can be seen, the image effect of the present scheme is that the fusion effect is like “injecting” an infrared thermal target into a visible light image, so that the details and contrast of the visible light image can be better preserved, and the infrared thermal target can be highlighted.

[0171] As shown in FIG. 10, (a) of FIG. 10 is a gray-scale fusion image obtained in some embodiments of the present application, that is, a base layer fusion image obtained by performing image fusion on the visible light base layer and the infrared base layer according to the infrared pixel weight curve and the visible light pixel weight curve, and then obtaining a gray-scale fusion image; (b) of FIG. 10 is a gray-scale fusion image obtained by performing average fusion on the visible light base layer and the infrared base layer.

[0172] As can be seen from the comparison between Fig. 10(a) and Fig. 10(b), the infrared thermal target is highlighted in Fig. 10(a), for example, the kettle in Fig. 10(a) can be clearly distinguished from the background, and other details in Fig. 10(a) can also be retained, while in Fig. 10(b), the traditional average fusion method is adopted, which is prone to insufficient contrast of the target image, especially for the target image with opposite light and shade in the visible light base layer and the infrared base layer, for example, the black kettle in Fig. 10(b) is placed in front of the white background, and due to its high temperature, it is brighter than the background in the infrared image, and thus after the traditional average fusion method is adopted, the contrast of the fusion image is low, and the thermal target cannot be highlighted.

[0173] Therefore, by using the above scheme, the visible light original image is subjected to N-layer pyramid decomposition to obtain a visible light base layer, the infrared original image is subjected to N-layer pyramid decomposition to obtain an infrared base layer, and the visible light base layer and the infrared base layer are fused according to the infrared pixel weight curve and the visible light pixel weight curve, so that the display weight of different pixel values of the infrared base layer in the base layer fusion image is high or low, thereby achieving the effect of "injecting" the infrared base layer into the visible light base layer, which can effectively improve the highlighting degree of the infrared target after fusion, and furthermore, since only structural information is included in the visible light base layer and the infrared base layer, the highlighting display of the infrared base layer in the base layer fusion image can effectively avoid the introduction of noise information, and can effectively ensure the fusion effect of the visible light base layer and the infrared base layer.

[0174] It should be noted that the above-mentioned some embodiments are black and white fusion algorithms for the visible light original image and the infrared original image, that is, the obtained fusion image is a gray-scale fusion image, that is, the gray-scale fusion image is black and white. In other embodiments of the present application, colorization can also be performed based on the visible light original image and the infrared original image to obtain a color image.

[0175] Referring to Fig. 11, a flowchart of another embodiment of the image processing method of the present application is shown.

[0176] The same as the foregoing embodiments, the present application will not be described here. The difference from the foregoing embodiments is that in some embodiments of the present application, colorization is performed based on the visible light original image and the infrared original image to obtain a color image.

[0177] Step S1, N-layer pyramid decomposition is performed on the visible light original image to obtain a visible light base layer; then, step S2, N-layer pyramid decomposition is performed on the infrared original image to obtain an infrared base layer; then, step S4, image fusion is performed on the visible light base layer and the infrared base layer according to an infrared pixel weight curve and a visible light pixel weight curve to obtain a base layer fused image; then, step S8, image restoration is performed according to the base layer fused image to obtain a gray-scale fused image.

[0178] In the process of step S1, N-layer pyramid decomposition is performed on the visible light original image to obtain a visible light base layer, in the step of performing Laplacian pyramid decomposition on the visible light original image for the jth time, an m-1 visible Gaussian layer is obtained; in the process of step S2, N-layer pyramid decomposition is performed on the infrared original image to obtain an infrared base layer, in the step of performing Laplacian pyramid decomposition on the infrared original image for the jth time, an m-1 infrared Gaussian layer is obtained.

[0179] As shown in FIG. 11, in some embodiments of the present application, the image processing method further comprises: step S9, obtaining two chroma layers according to the m-1 visible Gaussian layer and the m-1 infrared Gaussian layer; step S10, performing m-1 times of up-sampling on the two chroma layers respectively to obtain two chroma channel images; and step S11, obtaining a color image according to the two chroma channel images and the gray-scale fused image.

[0180] In some embodiments, in the step S9 of obtaining two chroma layers according to the m-1 visible Gaussian layer and the m-1 infrared Gaussian layer, two chroma layers are obtained according to the N-1 visible Gaussian layer and the N-1 infrared Gaussian layer. The N-1 visible Gaussian layer and the N-1 infrared Gaussian layer are Gaussian layers obtained in the last time of performing the image decomposition step. The noise in the Gaussian layer obtained in the last time of performing the image decomposition step is less, so that the noise in the obtained chroma layer is less, and storage resources are saved.

[0181] In some specific embodiments, the image processing method further comprises: setting a pseudo-color parameter; and in the step S9 of obtaining two chroma layers according to the m-1 visible Gaussian layer and the m-1 infrared Gaussian layer, two chroma layers are obtained according to the m-1 visible Gaussian layer and the m-1 infrared Gaussian layer in combination with the pseudo-color parameter.

[0182] For example, in the step of setting a pseudo-color parameter, the pseudo-color parameter can be set according to a color style.

[0183] Specifically, the steps for setting pseudo-color parameters include: setting pseudo-color parameters corresponding to the (m-1)th visible Gaussian layer and the (m-1)th infrared Gaussian layer of one of the chroma layers; and setting pseudo-color parameters corresponding to the (m-1)th visible Gaussian layer and the (m-1)th infrared Gaussian layer of another chroma layer.

[0184] In the step of obtaining two chromaticity layers based on the (m-1)th visible Gaussian layer and the (m-1)th infrared Gaussian layer, combined with the pseudo-color parameters, two chromaticity layers are obtained. Specifically: U_L m-1 =G m-1 e_vi*q1-G m-1 e_ir*q2; V_L m-1 =G m-1 e_vi*q3-G m-1 e_ir*q4.

[0185] Among them, U_L m-1 V_L is the U-channel chromaticity layer corresponding to the (m-1)th visible Gaussian layer and the (m-1)th infrared Gaussian layer. m-1 G is the V channel chromaticity layer corresponding to the (m-1)th visible Gaussian layer and the (m-1)th infrared Gaussian layer. m-1 e_vi is the (m-1)th visible Gaussian layer, G m-1 e_ir is the (m-1)th infrared Gaussian layer, q1 is the pseudo-color parameter corresponding to the U-channel chroma layer and the (m-1)th visible Gaussian layer, q2 is the pseudo-color parameter corresponding to the U-channel chroma layer and the (m-1)th infrared Gaussian layer, q3 is the pseudo-color parameter corresponding to the V-channel chroma layer and the (m-1)th visible Gaussian layer, and q4 is the pseudo-color parameter corresponding to the V-channel chroma layer and the (m-1)th infrared Gaussian layer. In some embodiments of the example, in step S1, the step of performing N-layer pyramid decomposition on the original visible light image, the original visible light image is decomposed into a 3-layer pyramid; in step S1, the step of performing N-layer pyramid decomposition on the original infrared image, the original infrared image is decomposed into a 3-layer pyramid; in the step of obtaining 2 chroma layers, the obtained chroma layers are: U_L2 = G2e_vi*q1 - G2e_ir*q2; V_L2 = G2e_vi*q3 - G2e_ir*q4.

[0186] U_L2=q1*G2e_vi+q2*G2e_ir, V_L2=q3*G2e_vi+q4*G2e_ir, wherein U_L2 is a U channel chroma layer corresponding to the 2nd visible Gaussian layer and the 2nd infrared Gaussian layer, V_L2 is a V channel chroma layer corresponding to the 2nd visible Gaussian layer and the 2nd infrared Gaussian layer, G2e_vi is the 2nd visible Gaussian layer, G2e_ir is the 2nd infrared Gaussian layer, q1 is a pseudo-color parameter corresponding to the U channel chroma layer and the 2nd visible Gaussian layer, q2 is a pseudo-color parameter corresponding to the U channel chroma layer and the 2nd infrared Gaussian layer, q3 is a pseudo-color parameter corresponding to the V channel chroma layer and the 2nd visible Gaussian layer, and q4 is a pseudo-color parameter corresponding to the V channel chroma layer and the 2nd infrared Gaussian layer.

[0187] In some embodiments, step S10, 2 chroma layers are respectively subjected to m-1 times of up-sampling to obtain 2 chroma channel images.

[0188] Specifically, a U channel chroma layer U_L m-1 corresponding to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer is up-sampled to obtain U_L m-2 = sample_up(U_L m-1 ), wherein U_L m-2 is a U channel chroma layer corresponding to the m-2th visible Gaussian layer and the m-2th infrared Gaussian layer, and sample_up is an up-sampling process.

[0189] Specifically, a V channel chroma layer V_L m-1 corresponding to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer is up-sampled to obtain V_L m-2 = sample_up(V_L m-1 ), wherein V_L m-2 is a V channel chroma layer corresponding to the m-2th visible Gaussian layer and the m-2th infrared Gaussian layer, and sample_up is an up-sampling process.

[0190] The up-sampling results are taken as two chroma channel images, i.e., a U chroma channel image U L m-2out and a V chroma channel image V L m-2out , i.e., U L m-2out = U_L m-2 , and V L0out = V_L m-2 .

[0191] In other embodiments, after obtaining the U channel chroma layer U_L m-2 corresponding to the m-2th visible Gaussian layer and the m-2th infrared Gaussian layer, the U channel chroma layer U_L m-2up-sampling until a U channel chroma layer U_L0 corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer is obtained, U_L0 = sample_up(U_L1), wherein the 0th visible Gaussian layer and the 0th infrared Gaussian layer are the Gaussian layers obtained in the process of performing the image decomposition step for the first time, and U_L1 is a U channel chroma layer corresponding to the 1st visible Gaussian layer and the 1st infrared Gaussian layer.

[0192] In some other embodiments, after obtaining a V channel chroma layer V_L m-2 then, the V channel chroma layer V_L m-2 up-sampling until a V channel chroma layer V_L0 corresponding to the 0th visible Gaussian layer and the 0th infrared Gaussian layer is obtained, V_L0 = sample_up(V_L1), wherein the 0th visible Gaussian layer and the 0th infrared Gaussian layer are the Gaussian layers obtained in the process of performing the image decomposition step for the first time, and V_L1 is a V channel chroma layer corresponding to the 1st visible Gaussian layer and the 1st infrared Gaussian layer.

[0193] The up-sampling results are taken as two chroma channel images, which are respectively a U chroma channel image U L0out and a V chroma channel image V L0out , i.e., U L0out = U_L0, V L0out = V_L0.

[0194] In some embodiments of the example, in the step S1 of performing N-layer pyramid decomposition on the visible light original image, 3-layer pyramid decomposition is performed on the visible light original image; in the step S1 of performing N-layer pyramid decomposition on the infrared original image, 3-layer pyramid decomposition is performed on the infrared original image; and in the step S10 of performing m-1 times of up-sampling on the two chroma layers respectively to obtain two chroma channel images, 2 times of up-sampling are performed on the two chroma layers respectively to obtain two chroma channel images.

[0195] The U channel chroma layer U_L2 corresponding to the second visible Gaussian layer and the second infrared Gaussian layer is up-sampled to obtain a U channel chroma channel image U_L1 corresponding to the first visible Gaussian layer and the first infrared Gaussian layer; the U channel chroma channel image U_L1 corresponding to the first visible Gaussian layer and the first infrared Gaussian layer is up-sampled to obtain a U channel chroma channel image U_L0 corresponding to the zeroth visible Gaussian layer and the zeroth infrared Gaussian layer; the V channel chroma channel image V_L2 corresponding to the second visible Gaussian layer and the second infrared Gaussian layer is up-sampled to obtain a V channel chroma channel image V_L1 corresponding to the first visible Gaussian layer and the first infrared Gaussian layer; the V channel chroma channel image V_L1 corresponding to the first visible Gaussian layer and the first infrared Gaussian layer is up-sampled to obtain a V channel chroma channel image V_L0 corresponding to the zeroth visible Gaussian layer and the zeroth infrared Gaussian layer. The specific up-sampling process is as follows: U_L1 = sample_up(U_L2); U_L0 = sample_up(U_L1); V_L1 = sample_up(V_L2); V_L0 = sample_up(V_L1).

[0196] wherein U_L2 is the U channel chroma layer corresponding to the second visible Gaussian layer and the second infrared Gaussian layer, U_L1 is the U channel chroma channel image corresponding to the first visible Gaussian layer and the first infrared Gaussian layer, U_L0 is the U channel chroma channel image corresponding to the zeroth visible Gaussian layer and the zeroth infrared Gaussian layer, V_L1 is the V channel chroma channel image corresponding to the first visible Gaussian layer and the first infrared Gaussian layer, V_L0 is the V channel chroma channel image corresponding to the zeroth visible Gaussian layer and the zeroth infrared Gaussian layer, and sample_up is the up-sampling processing.

[0197] The up-sampling results are two chroma channel images, which are a U chroma channel image U L0out and a V chroma channel image V L0out , i.e., U L0out = U_L0 and V L0out = V_L0.

[0198] In some embodiments, step S11, a color image is obtained according to the two chroma channel images and the grayscale fusion image.

[0199] Specifically, the image processing method further comprises: taking the grayscale fusion image as a luminance channel image; taking the above sampling results as two chroma channel images; and step S11, a color image is obtained according to the two chroma channel images and the grayscale fusion image, wherein a color image is obtained according to the two chroma channel images and the luminance channel image.

[0200] In some embodiments as shown in FIG. 4, in the step S1 of performing N-layer pyramid decomposition on the visible light original image, 3-layer pyramid decomposition is performed on the visible light original image; in the step S1 of performing N-layer pyramid decomposition on the infrared original image, 3-layer pyramid decomposition is performed on the infrared original image; in the step S11, a color image is obtained according to the two color channel images and the gray-scale fusion image, and the color image is obtained according to the U color channel image U L0out and the V color channel image V L0out and the luminance channel image Y L0out in the YUV space.

[0201] Specifically, in the step S8, after the image restoration is performed on the base layer fusion image, the 0th layer restored image GR0 is obtained, where the 0th layer restored image GR0 is the gray-scale fusion image, i.e., Y L0out = GR0.

[0202] Referring to FIG. 12 in combination with FIG. 4, compared with the gray-scale fusion image in FIG. 4, the color images in (a) of FIG. 12, (b) of FIG. 12 and (c) of FIG. 12 are all color images, which expand the application scenarios of image fusion, and it can be known from the comparison among (a) of FIG. 12, (b) of FIG. 12 and (c) of FIG. 12 that different color styles of color images are obtained through the adjustment of the pseudo-color parameters, so that the obtained color images have a natural feeling.

[0203] Still referring to FIG. 1, in some embodiments of the present application, before the step S3, there is further a step S30 of performing clipping operations on the ath visible light detail layer and the ath infrared detail layer respectively.

[0204] Referring to FIG. 13, a flowchart of another embodiment of the image processing method of the present application is shown.

[0205] The same as the foregoing embodiments, the present application will not be described here again. The difference from the foregoing embodiments is that in some embodiments of the present application, the image processing method can further perform a clipping operation on the obtained detail layer to limit the size of the pixel value of the obtained detail layer, so as to alleviate the halo effect.

[0206] The halo effect (Halo Effect) represents an optical defect on an image; it usually appears at the edge where the light and dark change intensively.

[0207] In some embodiments of the present application, the N-layer pyramid decomposition is performed on the visible light original image, and N-layer visible light detail layers are further obtained; the N-layer pyramid decomposition is performed on the infrared original image, and N-layer infrared detail layers are further obtained; where N is greater than or equal to 1; the image processing method further includes a step S30 of performing clipping operations on the ath visible light detail layer and the ath infrared detail layer respectively.

[0208] As shown in FIG. 13, in some embodiments of the present application, the image processing method comprises: step S1: performing N-layer pyramid decomposition on the visible light original image, and obtaining N visible light detail layers; then, step S301: performing pinch-off operation on the ath visible light detail layer; step S2: performing N-layer pyramid decomposition on the infrared original image, and obtaining N infrared detail layers; then, step S302: performing pinch-off operation on the ath infrared detail layer; then, step S303: performing image fusion on the pinch-off operated ath visible light detail layer and the pinch-off operated ath infrared detail layer, to obtain the ath detail fusion image.

[0209] Specifically, the pinch-off operation is to set the upper threshold and lower threshold of the pixel value of the visible light detail layer, and set the upper threshold and lower threshold of the pixel value of the infrared detail layer. In some embodiments, the pixel value range of the visible light detail layer is [-256, 255]; the pixel value range of the infrared detail layer is [-256, 255].

[0210] In some embodiments of the example, with reference to FIG. 4, in the step S1 of performing N-layer pyramid decomposition on the visible light original image, 3-layer pyramid decomposition is performed on the visible light original image; in the step S1 of performing N-layer pyramid decomposition on the infrared original image, 3-layer pyramid decomposition is performed on the infrared original image; the 0th Laplacian layer L0, the 1st Laplacian layer L1, and the 2nd Laplacian layer L2 are obtained, the Laplacian layers in the left Laplacian pyramid in FIG. 4 correspond to the 0th visible light detail layer, the 1st visible light detail layer, and the 2nd visible light detail layer, and the Laplacian layers in the right Laplacian pyramid in FIG. 4 correspond to the 0th visible light detail layer, the 1st visible light detail layer, and the 2nd visible light detail layer; the upper threshold and lower threshold of the pixel value of the 0th visible light detail layer are set; the upper threshold and lower threshold of the pixel value of the 1st visible light detail layer are set; the upper threshold and lower threshold of the pixel value of the 2nd visible light detail layer are set; the upper threshold and lower threshold of the pixel value of the 0th infrared detail layer are set; the upper threshold and lower threshold of the pixel value of the 1st infrared detail layer are set; and the upper threshold and lower threshold of the pixel value of the 2nd infrared detail layer are set.

[0211] As shown in FIG. 14, in some embodiments of the present application, the gray-scale fusion image obtained without using the pinch-off operation is shown in (a) of FIG. 14, and the gray-scale fusion image obtained by using the pinch-off operation is shown in (b) of FIG. 14. It can be seen that the halo of the edge of the kettle in (b) of FIG. 14 is less, and the edge is sharper.

[0212] It can be known from comparing the (a) of FIG. 14 and the (b) of FIG. 14 that after the pinch operation is performed on the visible light detail layer and the infrared detail layer, the gray-scale fusion image obtained by image fusion can inhibit the influence of the halo effect and reduce resource occupation.

[0213] Referring to FIG. 15, a schematic diagram of another embodiment of the image processing method of the present application is shown.

[0214] The same as the foregoing embodiments, the present application will not be described here. The difference from the foregoing embodiments is that in some embodiments of the present application, the number of endpoints of the infrared pixel weight curve can be greater than 4.

[0215] As shown in FIG. 15, the acquisition of the infrared pixel weight curve shown in FIG. 15 is specifically connecting the first endpoint A1 and the second endpoint A2, the second endpoint A2 and the third endpoint A3 to obtain the infrared pixel weight curve of the cold target region.

[0216] Connecting the third endpoint A3 and the fourth endpoint A4 obtains the infrared pixel weight curve of the background target region.

[0217] Connecting the fourth endpoint A4 and the fifth endpoint A5, the fifth endpoint A5 and the sixth endpoint A6 obtains the infrared pixel weight curve of the hot target region.

[0218] Referring to FIG. 16, the (a) of FIG. 16 is a gray-scale fusion image obtained according to the infrared pixel weight curve determined by four endpoints, and the (b) of FIG. 16 is a gray-scale fusion image obtained according to the infrared pixel weight curve determined by six endpoints.

[0219] Comparing the (a) of FIG. 16 and the (b) of FIG. 16, the method of determining the infrared pixel weight curve according to six endpoints not only improves the fusion ratio of the infrared base layer, but also makes the infrared pixel weight curve adjustment more fine, and the infrared hot target in the obtained gray-scale fusion image is more prominent, for example, the water kettle with higher temperature in the (b) of FIG. 16 is more prominent in the fused image.

[0220] Correspondingly, the present application also provides an image processing device, comprising a memory and a processor, the memory stores a computer program executable on the processor, and when the processor executes the program, the processor executes the steps of the image processing method described above.

[0221] In the image processing device, the processor executes the computer program stored on the memory to execute the steps of the image processing method of the present application, and the specific technical solutions of the image processing device can refer to the embodiments of the foregoing image processing method, and the present application will not be described here.

[0222] Correspondingly, the present application also provides a computer readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, and computer instructions are stored on the computer readable storage medium, and the computer instructions execute the steps of the image processing method described above when running.

[0223] The computer readable storage medium stores computer instructions that execute the steps of the image processing method of the present application when running, and the specific technical solutions of the computer readable storage medium refer to the embodiments of the image processing method described above, which will not be described here again.

[0224] In summary, the technical scheme of the present application obtains a visible light detail layer and an infrared detail layer by performing N-layer image decomposition on the visible light original image and the infrared original image, and performs image fusion on the a-th visible light detail layer and the a-th infrared detail layer according to the a-th detail fusion weight. Since the visible light detail layer and the infrared detail layer include noise information, the calculation of the detail fusion weight can adjust the noise in the visible light detail layer and the infrared detail layer, so that the image fusion using the detail fusion weight can reduce the noise in the visible light detail layer and the infrared detail layer, thereby ensuring the fusion effect of the visible light detail layer and the infrared detail layer.

[0225] Moreover, the technical scheme of the present application limits the infrared pixel values in the infrared detail layer and the visible light pixel values in the visible light detail layer by threshold limiting processing, so that the pixel values in the infrared detail layer and the visible light detail layer are limited, resource occupation is reduced, and the generation of the halo effect is inhibited.

[0226] In addition, the technical scheme of the present application obtains a visible light base layer by performing N-layer pyramid decomposition on the visible light original image, obtains an infrared base layer by performing N-layer pyramid decomposition on the infrared original image, and fuses the visible light base layer and the infrared base layer according to the infrared pixel weight curve and the visible light pixel weight curve, so that the display weight of different pixel values of the infrared base layer in the base layer fusion image is high or low, thereby achieving the effect of "injecting" the infrared base layer into the visible light base layer, which can effectively improve the prominence of the infrared target after fusion. Moreover, since only structure information is included in the visible light base layer and the infrared base layer, the prominent display of the infrared base layer in the base layer fusion image can effectively avoid introducing noise information, and can effectively ensure the fusion effect of the visible light base layer and the infrared base layer.

[0227] In addition, by setting the background target field, the hot target field and the cold target field in the infrared base layer pixel value range, and the infrared pixel weight corresponding to the infrared pixel value in different target fields being different, the display weight of the infrared pixel value in different target fields in the base layer fusion image is high or low, thereby ensuring that the infrared base layer can be highlighted in the base layer fusion image; and the infrared pixel weight corresponding to the infrared pixel value in the cold target field and the infrared pixel weight corresponding to the infrared pixel value in the hot target field are both greater than the infrared pixel weight corresponding to the infrared pixel value in the background target field, which can effectively improve the contrast and naturalness of the fused image, and effectively improve the fusion effect.

[0228] In addition, by setting the initial weight value corresponding to the minimum infrared pixel value, the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value, that is, setting different initial infrared pixel weights, and according to the different initial infrared pixel weights, the infrared pixel weight curve in different target fields is obtained, the infrared pixel weight corresponding to different infrared pixel values can be adjusted subsequently to obtain different infrared weight curves, the optimization of the infrared weight curve is realized, and the infrared base layer can be highlighted accurately in the base layer fusion image.

[0229] In addition, the technical scheme of the present application processes the color components of the visible light Gaussian up-sampling image, the infrared Gaussian up-sampling image and the gray fusion image after image decomposition to obtain a color output image, thereby expanding the application scenarios of image fusion.

[0230] Although the present application is disclosed as above, the present application is not limited thereto. Any person skilled in the art, without departing from the spirit and scope of the present application, can make various changes and modifications, and therefore the protection scope of the present application should be subject to the range defined by the claims.

Claims

1. An image processing method, characterized by, The method comprises the following steps: performing N-layer pyramid decomposition on the visible light original image to obtain N visible light detail layers; performing N-layer pyramid decomposition on the infrared original image to obtain N infrared detail layers, wherein N is greater than or equal to 1; performing image fusion on the a th visible light detail layer and the a th infrared detail layer according to the a th detail fusion weight to obtain an a th detail fusion image, wherein a is a positive integer greater than or equal to 0 and less than or equal to N-1.

2. The image processing method of claim 1, wherein, The step of obtaining the a th detail fusion image comprises: According to the a-th detail fusion weight, each infrared pixel value in the infrared detail layer, and each visible light pixel value in the visible light detail layer, a detail layer fusion image Lar = Lav*weight Lavi + Lai*(1-weight Lavi ) is calculated, where Lar represents the detail layer fusion image, weight Lavi represents the a-th detail fusion weight, Lav is the a-th visible light detail layer, and Lai is the a-th infrared detail layer.

3. The image processing method of claim 1, wherein, The step of obtaining the a th detail fusion image further comprises obtaining the a th detail fusion weight.

4. The image processing method of claim 3, wherein, The step of obtaining the a th detail fusion weight comprises: obtaining the activity of the a th infrared detail layer according to the a th infrared detail layer; obtaining the activity of the a th visible light detail layer according to the a th visible light detail layer; obtaining the a th detail fusion weight according to the activity of the a th infrared detail layer and the activity of the a th visible light detail layer, and combining the a th infrared noise reduction parameter and the a th visible light noise reduction parameter.

5. The image processing method of claim 1, wherein, The method further comprises: performing pinch-off operation on the a th visible light detail layer and the a th infrared detail layer before performing image fusion on the a th visible light detail layer and the a th infrared detail layer to obtain the a th detail fusion image; In the step of obtaining the a th detail fusion image, the a th visible light detail layer and the a th infrared detail layer subjected to the pinch-off operation are fused to obtain the a th detail fusion image.

6. The image processing method of claim 1, wherein, The method further comprises: performing N-layer pyramid decomposition on the visible light original image to obtain a visible light base layer; performing N-layer pyramid decomposition on the infrared original image to obtain an infrared base layer, wherein N is a natural number; 7. The image processing method of claim 6, wherein, The method further comprises:

8. The image processing method of claim 6, wherein, performing image fusion on the visible light base layer and the infrared base layer to obtain a base layer fusion image.

9. The image processing method of claim 8, wherein, In the step of performing image fusion on the visible light base layer and the infrared base layer to obtain the base layer fusion image, the visible light base layer and the infrared base layer are averaged to obtain the base layer fusion image. In the step of performing image fusion on the visible light base layer and the infrared base layer to obtain the base layer fusion image, the visible light base layer and the infrared base layer are fused according to an infrared pixel weight curve and a visible light pixel weight curve to obtain the base layer fusion image. The method further comprises: setting the infrared pixel weight curve; The step of setting the infrared pixel weight curve comprises: setting a background target region, a hot target region and a cold target region in the infrared base layer pixel value range, wherein the infrared pixel value in the background target region is between the infrared pixel value in the hot target region and the infrared pixel value in the cold target region; The infrared pixel weight curve of the background target field, the infrared pixel weight curve of the hot target field and the infrared pixel weight curve of the cold target field are respectively set, wherein the infrared pixel weight corresponding to the infrared pixel value in the background target field is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the hot target field; the infrared pixel weight corresponding to the infrared pixel value in the background target field is less than or equal to the infrared pixel weight corresponding to the infrared pixel value in the cold target field.

10. The image processing method of claim 9, wherein, In the step of setting the background target field, the hot target field and the cold target field in the infrared base layer pixel value range, the average value of the infrared base layer pixel value is located in the background target field.

11. The image processing method of claim 9, wherein, The initial infrared pixel weight corresponding to the minimum infrared pixel value, the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value are respectively set, wherein the minimum infrared pixel value, the first infrared pixel value, the second infrared pixel value and the maximum infrared pixel value increase in turn, the initial infrared pixel weight corresponding to the minimum infrared pixel value is greater than or equal to the initial infrared pixel weight corresponding to the first infrared pixel value, the initial infrared pixel weight corresponding to the maximum infrared pixel value is greater than or equal to the initial infrared pixel weight corresponding to the second infrared pixel value, and the minimum infrared pixel value and the first infrared pixel value are the cold target field, the first infrared pixel value and the second infrared pixel value are the background target field, and the second infrared pixel value and the maximum infrared pixel value are the hot target field; The infrared pixel weight curve of the cold target field is obtained according to the initial infrared pixel weight corresponding to the minimum infrared pixel value and the initial infrared pixel weight corresponding to the first infrared pixel value; The infrared pixel weight curve of the background target field is obtained according to the initial infrared pixel weight corresponding to the first infrared pixel value and the initial infrared pixel weight corresponding to the second infrared pixel value; The infrared pixel weight curve of the hot target field is obtained according to the initial infrared pixel weight corresponding to the second infrared pixel value and the initial infrared pixel weight corresponding to the maximum infrared pixel value.

12. The image processing method of claim 9, wherein, The step of obtaining the base layer fusion image comprises: The base layer fusion image Gnr is calculated according to the infrared pixel weight curve, the visible light pixel weight curve, each visible light pixel value in the visible light base layer and each infrared pixel value in the infrared base layer, wherein Gnr represents the base layer fusion image, w_vis represents the visible light pixel weight, w_ir represents the infrared pixel weight, Gnv represents the visible light pixel value, Gni represents the infrared pixel value, and n is a natural number.

13. The image processing method of claim 1, wherein, The image processing method further comprises: performing image restoration on the 0th detail fusion image,..., the ath detail fusion image,..., and the (N-1)th detail fusion image to obtain a gray-scale fusion image.

14. The image processing method of claim 13, wherein, The visible light original image is subjected to N-layer pyramid decomposition, and a visible light base layer is further obtained. The infrared original image is decomposed into N layers of pyramids to obtain an infrared base layer; The image processing method further comprises: In the step of obtaining the gray fused image, image restoration is performed according to the base layer fused image, the 0th detail fused image, the ath detail fused image, the N-1th detail fused image to obtain the gray fused image.

15. The image processing method of claim 1, wherein, In at least one of the step of decomposing the visible light original image into N layers of pyramids and the step of decomposing the infrared original image into N layers of pyramids, the image to be processed is decomposed into N layers of Laplacian pyramids. The image to be processed is one of the visible light original image and the infrared original image.

16. The image processing method of claim 15, wherein, The step of decomposing into N layers of Laplacian pyramids comprises: performing the step of decomposing into Laplacian pyramids at least once, performing the step of decomposing into Laplacian pyramids the jth time to obtain a j-1th Laplacian layer, j being a positive integer greater than or equal to 0 and less than or equal to N. The step of performing the step of decomposing into Laplacian pyramids the mth time comprises: performing Gaussian filtering on the m-1th layer image; performing 1 / 2 down-sampling on the m-1th layer image to obtain a sampled and filtered mth layer image; performing up-sampling on the mth layer image to obtain an m-1th Gaussian layer; obtaining an m-1th Laplacian layer according to the m-1th Gaussian layer and the m-1th layer image, m being a positive integer greater than or equal to 2 and less than or equal to N.

17. The image processing method of claim 16, wherein, In the step of performing the step of decomposing into Laplacian pyramids the first time, the image to be processed is Gaussian filtered to obtain a 0th Laplacian layer.

18. The image processing method of claim 16, wherein, The image processing method further comprises: decomposing the visible light original image into N layers of pyramids to obtain a visible light base layer; decomposing the infrared original image into N layers of pyramids to obtain an infrared base layer, N being a natural number; and performing image fusion on the visible light base layer and the infrared base layer to obtain a base layer fused image. Performing image restoration according to the base layer fused image to obtain a gray fused image. Obtaining two chroma layers according to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer. Performing m-1th up-sampling on the two chroma layers respectively to obtain two chroma channel images. Obtaining a color image according to the two chroma channel images and the gray fused image.

19. The image processing method of claim 18, wherein, In the step of obtaining two chroma layers according to the m-1th visible Gaussian layer and the m-1th infrared Gaussian layer, two chroma layers are obtained according to the N-1th visible Gaussian layer and the N-1th infrared Gaussian layer.

20. An image processing apparatus characterized by comprising: comprises: a memory and a processor, the memory storing a computer program executable on the processor, and the processor executing the program to perform the steps of the image processing method of any one of claims 1 to 19.

21. A computer-readable storage medium, characterized in that, The computer readable storage medium is a non-volatile storage medium or a non-transient storage medium, and computer instructions are stored thereon, and the computer instructions are executed to perform the steps of the image processing method of any one of claims 1 to 19.

Images