An image fusion method, device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose an image fusion method and device, electronic equipment and storage medium, the method comprising: collecting n frames of to-be-processed images, n being a positive integer; calculating Kalman gain value and actual pixel value of the K-n+1th to-be-processed image according to initial prediction deviation and initial measurement deviation, predicted pixel value and measured pixel value of the K-n+1th to-be-processed image, K being a positive integer greater than or equal to n; recursively calculating the Kalman gain value of the K-n+1th to-be-processed image to obtain the Kalman gain value and actual pixel value of the K-n+2th to-be-processed image, and iterating until the Kalman gain value and actual pixel value of the Kth to-be-processed image are obtained; performing fusion processing on the n frames of to-be-processed images according to the actual pixel value of the n frames of to-be-processed images to obtain a fused image; and using the Kalman gain value of the previous frame of image to recursively and iteratively remove image noise and improve the smoothness and quality of the fused image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image fusion method and device, electronic equipment and storage medium. BACKGROUND

[0002] At present, image sensors are involved in some application scenarios of devices such as vehicles, smart phones, unmanned aerial vehicles and robots. In some specific scenarios, a plurality of images collected by the sensors may need to be fused and spliced to obtain image information with specific value.

[0003] However, in actual application, it is found that factors such as motion, shaking or light will interfere with image collection, affect image quality, and thus reduce the quality of the fused image. SUMMARY

[0004] The embodiments of the present application disclose an image fusion method and device, electronic equipment and storage medium, which can effectively remove image noise, reduce the interference of factors such as light, motion and shaking on images, reduce errors, and improve the smoothness and quality of the fused image.

[0005] The first aspect of the embodiments of the present application discloses an image fusion method, which can include:

[0006] Collecting n frames of to-be-processed images, n being a positive integer;

[0007] According to the initial prediction deviation and the initial measurement deviation, the predicted pixel value and the measured pixel value of the K-n+1th to-be-processed image, calculating the Kalman gain value and the actual pixel value of the K-n+1th to-be-processed image, K being a positive integer greater than or equal to n;

[0008] According to the Kalman gain value of the K-n+1th to-be-processed image, recursively calculating to obtain the Kalman gain value and the actual pixel value of the K-n+2th to-be-processed image, and iterating until the Kalman gain value and the actual pixel value of the Kth to-be-processed image are obtained;

[0009] According to the actual pixel value of the n frames of to-be-processed images, performing fusion processing on the n frames of to-be-processed images to obtain a fused image.

[0010] As an optional implementation, in the first aspect of the embodiments of the present application, the calculating the Kalman gain value and the actual pixel value of the K-n+1th to-be-processed image according to the initial prediction deviation and the initial measurement deviation, the predicted pixel value and the measured pixel value of the K-n+1th to-be-processed image includes:

[0011] Calculating the covariance of the initial prediction deviation and the initial measurement deviation as the Kalman gain value of the K-n+1th to-be-processed image.

[0012] According to the Kalman gain value of the K-n+1th frame to-be-processed image, the predicted pixel value of the K-n+1th frame to-be-processed image and the measured pixel value, the actual pixel value of the K-n+1th frame to-be-processed image is calculated.

[0013] As an optional implementation, in the first aspect of the embodiment of the present application, the recursive calculation according to the Kalman gain value of the K-n+1th frame to-be-processed image to obtain the Kalman gain value and the actual pixel value of the K-n+2th frame to-be-processed image comprises:

[0014] According to the Kalman gain value of the K-n+1th frame to-be-processed image and the initial prediction deviation, the optimized deviation of the K-n+1th frame to-be-processed image is calculated and obtained;

[0015] According to the optimized deviation of the K-n+1th frame to-be-processed image, the current prediction deviation of the K-n+2th frame to-be-processed image is calculated and obtained;

[0016] The covariance of the current prediction deviation of the K-n+2th frame to-be-processed image and the initial measurement deviation is calculated as the Kalman gain value of the K-n+2th frame to-be-processed image;

[0017] According to the Kalman gain value of the K-n+2th frame to-be-processed image, the predicted pixel value of the K-n+2th frame to-be-processed image and the measured pixel value, the actual pixel value of the K-n+2th frame to-be-processed image is calculated.

[0018] As an optional implementation, in the first aspect of the embodiment of the present application, the iteration until the Kalman gain value and the actual pixel value of the Kth frame to-be-processed image are obtained comprises:

[0019] When K-n+2 is equal to K, the step of performing the fusion processing on the n frames to-be-processed image according to the actual pixel value of the n frames to-be-processed image to obtain a fusion image is executed;

[0020] When K-n+2 is less than K, the iteration until the Kalman gain value and the actual pixel value of the Kth frame to-be-processed image are obtained.

[0021] As an optional implementation, in the first aspect of the embodiment of the present application, the fusion processing on the n frames to-be-processed image according to the actual pixel value of the n frames to-be-processed image to obtain a fusion image comprises:

[0022] According to the actual pixel value of the Kth frame to-be-processed image and the actual pixel value of the K-1th frame to-be-processed image, the Kth frame to-be-processed image and the K-1th frame to-be-processed image are fused, and the fusion is iterated until the K-n+1th frame to-be-processed image is fused, so that the fused image is obtained.

[0023] The second aspect of the embodiment of the present application discloses an image fusion device, which can include:

[0024] The acquisition module is configured to acquire n frames of to-be-processed images, where n is a positive integer.

[0025] The optimization module is configured to calculate the Kalman gain value and the actual pixel value of the K-n+1th frame to-be-processed image according to the initial prediction deviation and the initial measurement deviation, the prediction pixel value and the measurement pixel value of the K-n+1th frame to-be-processed image, where K is a positive integer greater than or equal to n; and recursively calculate the Kalman gain value and the actual pixel value of the K-n+2th frame to-be-processed image according to the Kalman gain value of the K-n+1th frame to-be-processed image, and iterate until the Kalman gain value and the actual pixel value of the Kth frame to-be-processed image are obtained.

[0026] The fusion module is configured to fuse the n frames of to-be-processed images according to the actual pixel values of the n frames of to-be-processed images, and obtain a fused image.

[0027] As an optional implementation, in the second aspect of the embodiment of the present application, the optimization module is configured to calculate the Kalman gain value and the actual pixel value of the K-n+1th frame to-be-processed image according to the initial prediction deviation and the initial measurement deviation, the prediction pixel value and the measurement pixel value of the K-n+1th frame to-be-processed image in the following manner:

[0028] Calculate the covariance of the initial prediction deviation and the initial measurement deviation as the Kalman gain value of the K-n+1th frame to-be-processed image.

[0029] Calculate the actual pixel value of the K-n+1th frame to-be-processed image according to the Kalman gain value of the K-n+1th frame to-be-processed image, the prediction pixel value and the measurement pixel value of the K-n+1th frame to-be-processed image.

[0030] As an optional implementation, in the second aspect of the embodiment of the present application, the optimization module is configured to recursively calculate the Kalman gain value and the actual pixel value of the K-n+2th frame to-be-processed image according to the Kalman gain value of the K-n+1th frame to-be-processed image in the following manner:

[0031] According to the Kalman gain value of the K-n+1th frame to-be-processed image and the initial prediction deviation, the optimization deviation of the K-n+1th frame to-be-processed image is calculated.

[0032] According to the optimized deviation of the K-n+1th frame to-be-processed image, a current prediction deviation of the K-n+2th frame to-be-processed image is obtained;

[0033] A covariance of the current prediction deviation of the K-n+2th frame to-be-processed image and the initial measurement deviation is calculated as a Kalman gain value of the K-n+2th frame to-be-processed image;

[0034] According to the Kalman gain value of the K-n+2th frame to-be-processed image, the predicted pixel value and the measurement pixel value of the K-n+2th frame to-be-processed image, an actual pixel value of the K-n+2th frame to-be-processed image is calculated.

[0035] As an optional implementation, in the second aspect of the embodiment of the present application, the manner in which the optimization module is used to iterate until the Kalman gain value and the actual pixel value of the Kth frame to-be-processed image are obtained is specifically as follows:

[0036] When K-n+2 is equal to K, the step of performing fusion processing on the n frames to-be-processed images according to the actual pixel value of the n frames to-be-processed images to obtain a fusion image is executed;

[0037] When K-n+2 is less than K, iteration is performed until the Kalman gain value and the actual pixel value of the Kth frame to-be-processed image are obtained.

[0038] As an optional implementation, in the second aspect of the embodiment of the present application, the manner in which the fusion module is used to perform fusion processing on the n frames to-be-processed images according to the actual pixel value of the n frames to-be-processed images to obtain a fusion image is specifically as follows:

[0039] According to the actual pixel value of the Kth frame to-be-processed image and the actual pixel value of the K-1th frame to-be-processed image, fusion processing is performed on the Kth frame to-be-processed image and the K-1th frame to-be-processed image, and iteration is performed until the K-n+1th frame to-be-processed image is fused to obtain the fusion image.

[0040] The third aspect of the embodiment of the present application discloses an electronic device, which can include:

[0041] a memory in which executable program codes are stored;

[0042] a processor coupled with the memory;

[0043] The processor invokes the executable program codes stored in the memory to execute the image fusion method disclosed in the first aspect of the embodiment of the present application.

[0044] The fourth aspect of the embodiments of the present application discloses a computer readable storage medium storing a computer program, wherein the computer program causes a computer to execute the image fusion method disclosed in the first aspect of the embodiments of the present application.

[0045] The fifth aspect of the embodiments of the present application discloses a computer program product, which causes a computer to execute part or all steps of any method of the first aspect when the computer program product runs on the computer.

[0046] The sixth aspect of the embodiments of the present application discloses an application publishing platform for publishing a computer program product, wherein the computer program product causes a computer to execute part or all steps of any method of the first aspect when the computer program product runs on the computer.

[0047] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0048] In the embodiments of the present application, n frames of to-be-processed images are collected, and the Kalman gain value and the actual pixel value of the K-n+1th to-be-processed image are calculated according to the initial prediction deviation and the initial measurement deviation, the predicted pixel value and the measured pixel value of the K-n+1th to-be-processed image, wherein n is a positive integer, and K is a positive integer greater than or equal to n. Then, the Kalman gain value of the K-n+2th to-be-processed image is further calculated recursively according to the Kalman gain value of the K-n+1th to-be-processed image, and the iteration is performed until the Kalman gain value and the actual pixel value of the Kth to-be-processed image are obtained. Finally, the n frames of to-be-processed images are fused according to the actual pixel values of the n frames of to-be-processed images to obtain a fused image. As can be seen, by implementing the embodiments of the present application, the Kalman gain value of the previous image can be used for recursive and iterative calculation, so that the Kalman gain converges to the true value to improve the accuracy of the actual pixel value of the to-be-processed image, effectively remove image noise, reduce the interference of factors such as light, motion and jitter on the image, reduce errors, and improve the smoothness and quality of the fused image. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0050] Figure 1 The flowchart of the image fusion method disclosed in Embodiment One of the present application is shown in the figure.

[0051] Figure 2A flowchart of an image fusion method disclosed in Embodiment Two of the present application;

[0052] Figure 3 A flowchart of an image fusion method disclosed in Embodiment Three of the present application;

[0053] Figure 4 A structural diagram of an image fusion device disclosed in the present application;

[0054] Figure 5 A structural diagram of an electronic device disclosed in the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0056] It should be noted that the terms "first", "second", "third", and "fourth" in the specification and claims of the present application are used to distinguish different objects, rather than to describe a specific order. The terms "include" and "have" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or device.

[0057] The embodiments of the present application disclose an image fusion method, device, electronic device, and storage medium, which can effectively remove image noise, reduce the interference of factors such as light, motion, and jitter on images, reduce errors, and improve the smoothness and quality of the fused image.

[0058] The technical solutions of the present application will be described in detail below through specific embodiments.

[0059] Please refer to Figure 1 , Figure 1 A flowchart of an image fusion method disclosed in Embodiment One of the present application; as Figure 1 shown, the image fusion method can include:

[0060] 101, acquiring n frames of images to be processed, n being a positive integer.

[0061] The execution subject of the embodiments of the present application is an image fusion device or an electronic device. The image fusion device can be a device independent of the electronic device or a module built in the image fusion device. The electronic device can be a mobile terminal (for example, a smart phone), a wearable device (for example, a smart bracelet or a smart watch), a tablet computer, a vehicle-mounted device, or the like.

[0062] The electronic device starts the image sensor or other image acquisition device to acquire an image. In the embodiments of the present application, n+1 frames of low-noise images from the K-nth frame to the Kth frame are acquired from a series of frame images, and then the K-n+1th frame to the Kth frame are taken as the to-be-processed images. Meanwhile, the predicted pixel value of the K-n+1th frame can be obtained according to the K-nth frame.

[0063] 102. Calculate the Kalman gain value and the actual pixel value of the K-n+1th frame to-be-processed image according to the initial prediction deviation and the initial measurement deviation, the predicted pixel value and the measured pixel value of the K-n+1th frame to-be-processed image, K being a positive integer greater than or equal to n.

[0064] In the embodiments of the present application, the initial prediction deviation x1 of the Gaussian noise can be set in advance, which can be any estimate within a certain range. That is, the initial prediction deviation x1 is an estimate value determined within a certain range. The initial measurement deviation x2 of the Gaussian noise is also set in advance, which is the initial measurement deviation caused by the image sensor. For example, the initial prediction deviation x1 is set to 10, and the initial measurement deviation x2 is set to 5. Generally, the initial prediction deviation x1 is larger, and the initial measurement deviation x2 is smaller.

[0065] It should be noted that the initial prediction deviation x1 is initialized as the current prediction deviation of the K-n+1th frame. The initial measurement deviation x2 is initialized as the Gaussian noise deviation of the image sensor, which is generally unchanged in the entire calculation process. The initial prediction deviation x1 can be calculated according to the optimized deviation of the K-nth frame to-be-processed image and a constant. In the embodiments of the present application, the optimized deviation of the K-nth frame to-be-processed image can be set in advance, and then the initial prediction deviation x1 can be estimated according to the optimized deviation of the K-nth frame to-be-processed image and the constant. Then, the Kalman gain value and the optimized deviation of the K-n+1th frame to-be-processed image are calculated according to the initial prediction deviation x1 and the initial measurement deviation x2. Details will be described in Embodiment 2, and will not be described here.

[0066] The constant is the uncertainty of the prediction when the predicted pixel value is calculated, which can be set according to the experience value.

[0067] wherein, the prediction pixel value of the K-n+1 frame to-be-processed image at the same position point when photographing the same object should be equal to the prediction pixel value of the K-n frame low-noise image, the prediction pixel value of the K-n frame low-noise image is represented by I(x, y)(K-n), and the prediction pixel value of the K-n+1 frame to-be-processed image is represented by I(x, y)(K-n+1), and the following equation is obtained:

[0068] I(x, y)(K-n+1) = I(x, y)(K-n) (1)

[0069] The measurement pixel value of the K-n+1 frame to-be-processed image can be obtained according to the image. Then, according to the Kalman gain value of the K-n+1 frame to-be-processed image, the prediction pixel value and the measurement pixel value of the K-n+1 frame to-be-processed image, the actual pixel value of the K-n+1 frame to-be-processed image can be calculated, which will be described in detail in Embodiment 2.

[0070] 103. The Kalman gain value and the actual pixel value of the K-n+2 frame to-be-processed image are obtained by recursive calculation according to the Kalman gain value of the K-n+1 frame to-be-processed image, and iteration is performed until the Kalman gain value and the actual pixel value of the K frame to-be-processed image are obtained.

[0071] In step 103, recursive calculation is performed by using the Kalman gain value of the K-n+1 frame to-be-processed image to enter the calculation of the Kalman gain value and the actual pixel value of the next frame, i.e., the K-n+2 frame to-be-processed image.

[0072] The source of the prediction pixel value of the K-n+1 frame to-be-processed image is the same, and the prediction pixel value of the K-n+2 frame to-be-processed image is represented by I(x, y)(K-n+2), and the following equation is obtained:

[0073] I(x, y)(K-n+2) = I(x, y)(K-n+1) (2)

[0074] The measurement pixel value of the K-n+2 frame to-be-processed image can be obtained according to the image.

[0075] It should be noted that after the Kalman gain value of the K-n+2 frame to-be-processed image is calculated, recursive calculation is performed by using the Kalman gain value of the K-n+2 frame to-be-processed image to directly obtain the Kalman gain value and the actual pixel value of the K frame to-be-processed image, and the K frame to-be-processed image is the last frame in the n frames to-be-processed images.

[0076] If K-n+2 is equal to K, after the Kalman gain value and the actual pixel value of the K-n+2th frame of the to-be-processed image are calculated, step 104 is performed, otherwise, iteration is performed until the Kalman gain value and the actual pixel value of the K-n+2th frame of the to-be-processed image are obtained, and then step 104 is performed.

[0077] 104, fusing the n frames of to-be-processed images according to the actual pixel values of the n frames of to-be-processed images to obtain a fused image.

[0078] In step 104, the n frames of to-be-processed images are fused and spliced according to the actual pixel values corresponding to the K-n+1th frame of to-be-processed image to the Kth frame of to-be-processed image, to obtain a fused image.

[0079] In the embodiment of the present application, n frames of to-be-processed images are collected, the Kalman gain value and the actual pixel value of the K-n+1th frame of to-be-processed image are calculated according to the initial prediction deviation and the initial measurement deviation, the predicted pixel value and the measured pixel value of the K-n+1th frame of to-be-processed image, wherein n is a positive integer, and K is a positive integer greater than or equal to n, then the Kalman gain value of the K-n+1th frame of to-be-processed image is further recursively calculated to obtain the Kalman gain value and the actual pixel value of the K-n+2th frame of to-be-processed image, and iteration is performed until the Kalman gain value and the actual pixel value of the Kth frame of to-be-processed image are obtained, finally, the n frames of to-be-processed images are fused according to the actual pixel values of the n frames of to-be-processed images to obtain a fused image; it can be seen that, by implementing the embodiment of the present application, the Kalman gain value of the previous frame of image is used for continuous recursion and iteration, so that the Kalman gain continuously converges to approach the true value, the accuracy of the actual pixel value of the to-be-processed image is improved, image noise is effectively removed, the interference caused by factors such as light, motion and jitter on the image is reduced, errors are reduced, and the smoothness and quality of the fused image are improved.

[0080] Please refer to Figure 2 , Figure 2 The flowchart of the image fusion method disclosed in Embodiment Two of the present application is shown in FIG. 2. Figure 2 As shown in FIG. 2, the image fusion method can include the following steps.

[0081] 201, collecting n frames of to-be-processed images, wherein n is a positive integer.

[0082] The execution subject of the embodiment of the present application is an image fusion device or an electronic device, wherein the image fusion device can be a device independent of the electronic device before or a module built-in in the image fusion device. The electronic device can be a mobile terminal (for example, a smart phone), a wearable device (for example, a smart bracelet, a smart watch), a tablet computer, a vehicle-mounted device, etc.

[0083] 202、calculate the covariance of the initial prediction deviation and the initial measurement deviation as the Kalman gain value of the K-n+1 frame to-be-processed image.

[0084] wherein the initial prediction deviation x1 is obtained according to the optimized deviation of the K-n frame and the constant as introduced above, for example, W(K-n) represents the optimized deviation of the K-n frame, C is the constant, in the embodiment of the present application, the constant C can be set according to the experience value, and a prediction value is given to W(K-n), then the calculation formula of the initial prediction deviation x1 is as follows:

[0085] x1 = ((W(K-n))2+C2)0.5 (3)

[0086] that is, the square of W(K-n) is calculated, the square of the constant C is calculated, the sum of the two squares is further calculated, and then the square root of the sum is calculated.

[0087] In step 202, the calculation formula of the covariance of the initial prediction deviation x1 and the initial measurement deviation x2 is as follows:

[0088] Kg(K-n+1) = x1 2 / (x1 2+x2 2) (4)

[0089] wherein Kg(K-n+1) is the covariance, that is, the Kalman gain value of the K-n+1 frame to-be-processed image.

[0090] 203、according to the Kalman gain value of the K-n+1 frame to-be-processed image, the prediction pixel value and the measurement pixel value of the K-n+1 frame to-be-processed image, calculate the actual pixel value of the K-n+1 frame to-be-processed image.

[0091] In step 202, the Kalman gain value of the K-n+1 frame to-be-processed image is calculated, and then the calculation formula of the actual pixel value of the K-n+1 frame to-be-processed image is as follows:

[0092] I'(x, y)(K-n+1) = I(x, y)(K-n+1) + Kg(K-n+1)*(Y(K-n+1)-I(x, y)(K-n+1)) (5)

[0094] wherein I'(x, y)(K-n+1) represents the actual pixel value of the K-n+1 frame to-be-processed image, Y(K-n+1) represents the measurement pixel value of the K-n+1 frame to-be-processed image.

[0095] 204、according to the Kalman gain value of the K-n+1 frame to-be-processed image and the initial prediction deviation, calculate the optimized deviation of the K-n+1 frame to-be-processed image.

[0096] Further, before entering the K-n+2 frame to be processed image, it is also necessary to calculate the K-n+1 frame to be processed image optimization deviation, the calculation formula is as follows:

[0097] X0=(1-Kg(K-n+1))*x1^2)^0.5 (6)

[0098] Wherein, X0 indicates the K-n+1 frame to be processed image optimization deviation, x1 is the initial prediction deviation.

[0099] 205, according to the K-n+1 frame to be processed image optimization deviation calculation obtained K-n+2 frame to be processed image current prediction deviation.

[0100] Wherein, according to the above formula (3), can obtain the K-n+2 frame to be processed image current prediction deviation calculation formula as follows:

[0101] x1'(K-n+2)=(X0^2+C^2)^0.5) (7)

[0102] Wherein, X1'(K-n+2) is the K-n+2 frame to be processed image current prediction deviation, X0 is the K-n+1 frame to be processed image optimization deviation.

[0103] In formula (7), using X0, namely K-n+1 frame to be processed image optimization deviation instead of the original K-n frame to be processed image optimization deviation, obtains the K-n+2 frame to be processed image current prediction deviation.

[0104] 206, calculate the K-n+2 frame to be processed image current prediction deviation and the initial measurement deviation covariance, as the K-n+2 frame to be processed image Kalman gain value.

[0105] Wherein, according to the above formula (4), can obtain the K-n+2 frame to be processed image Kalman gain value calculation formula as follows:

[0106] Kg(K-n+2)=X1'(K-n+2)^2 / (X1'(K-n+2)^2+x2^2) (8)

[0107] In the above formula, although the initial measurement deviation x2 is constant, but because in the K-n+2 frame to be processed image, using the K-n+2 frame to be processed image current prediction deviation X1'(K-n+2) instead of the initial prediction deviation x1, has changed, so that the Kalman gain value is constantly convergent, further let the current prediction deviation of the following to be processed image is more and more close to the true value, so as to make the accuracy of the optimized actual pixel value is higher, obtain higher quality image.

[0108] 207. Calculate the actual pixel value of the K-n+2th to-be-processed image according to the Kalman gain value of the K-n+2th to-be-processed image, the predicted pixel value of the K-n+2th to-be-processed image and the measured pixel value.

[0109] wherein the predicted pixel value of the K-n+2th to-be-processed image is equal to the predicted pixel value of the K-n+1th to-be-processed image, i.e. I(x,y)(K-n+2) = I(x,y)(K-n+1).

[0110] Further, the calculation formula of the actual pixel value of the K-n+2th to-be-processed image is as follows:

[0111] I'(x,y)(K-n+2) = I(x,y)(K-n+2) + Kg(K-n+2)*(Y(K-n+2) - I(x,y)(K-n+2))(9)

[0112] 208. Iterate until the Kalman gain value and the actual pixel value of the Kth to-be-processed image are obtained.

[0113] wherein if K-n+2 = K, it is indicated that the actual pixel value of the n th to-be-processed image has been calculated, and step 209 is performed, otherwise, the Kalman gain value of the K-n+2th to-be-processed image is used for recursive calculation, the current predicted deviation of the K-n+3th to-be-processed image is obtained first, then the covariance of the current predicted deviation of the K-n+3th to-be-processed image and the initial measured deviation is calculated as the Kalman gain value of the K-n+3th to-be-processed image, and then the actual pixel value of the K-n+3th to-be-processed image is calculated according to the Kalman gain value of the K-n+3th to-be-processed image, the predicted pixel value of the K-n+3th to-be-processed image and the measured pixel value, and the iteration calculation is sequentially performed until the actual pixel value of the Kth to-be-processed image is obtained.

[0114] 209. Perform fusion processing on the n th to-be-processed image according to the actual pixel value of the n th to-be-processed image to obtain a fusion image.

[0115] By implementing the embodiment of the present application, the Kalman gain value of the previous image is used for recursive and iterative calculation, so that the Kalman gain converges to the real value to improve the accuracy of the actual pixel value of the to-be-processed image, effectively removes image noise, reduces the interference of factors such as light, motion and jitter on the image, reduces errors, and improves the smoothness and quality of the fusion image.

[0116] Please refer to Figure 3 , Figure 3 for the flowchart of the image fusion method disclosed in Embodiment Three of the present application; as shown in Figure 3 , the image fusion method can include:

[0117] 301, collect n frames of images to be processed, n is a positive integer.

[0118] The execution subject of the embodiment of the present application is an image fusion device or an electronic device. The image fusion device can be a device independent of the electronic device or a module built in the image fusion device. The electronic device can be a mobile terminal (for example, a smart phone), a wearable device (for example, a smart bracelet or a smart watch), a tablet computer, a vehicle-mounted device, etc.

[0119] 302, according to the initial prediction deviation and the initial measurement deviation, the predicted pixel value and the measured pixel value of the K-n+1 frame of image to be processed, calculate the Kalman gain value and the actual pixel value of the K-n+1 frame of image to be processed, K is a positive integer greater than or equal to n.

[0120] For more information about step 302, please refer to the introduction of step 102 above.

[0121] 303, according to the recursive calculation of the Kalman gain value of the K-n+1 frame of image to be processed, obtain the Kalman gain value and the actual pixel value of the K-n+2 frame of image to be processed, and iterate until the Kalman gain value and the actual pixel value of the K frame of image to be processed are obtained.

[0122] For more information about step 303, please refer to the introduction of step 103 above.

[0123] 304, according to the actual pixel value of the K frame of image to be processed and the actual pixel value of the K-1 frame of image to be processed, the K frame of image to be processed and the K-1 frame of image to be processed are fused, and the iteration is performed until the K-n+1 frame of image to be processed is fused, and the fused image is obtained.

[0124] In steps 301-303, the Kalman gain value is continuously converged to optimize the image to be processed and realize the denoising processing of the image to be processed. Then in step 304, starting from the K frame of image to be processed, the optimized pixel value is used to fuse the K frame of image to be processed and the K-1 frame of image to be processed to obtain a sub-fused image. The sub-fused image is fused with the K-2 frame of image to be processed, and so on, until the last sub-fused image is fused with the K-n+1 frame of image to be processed to obtain the final fused image. Since the denoising processing is performed on each frame of image to be processed, the pixel quality of each frame of image to be processed is improved, so that the fused quality of the fused image is also improved, and the smoothness is better, which is suitable for the application scene of panoramic stitching algorithm.

[0125] Optionally, the fusion method of step 304 can use a Gaussian pyramid algorithm for fusion, and can also use a traditional fusion algorithm (such as direct average, weighted average algorithm), optimal seam search, Perona fusion algorithm, etc. for fusion. Since the images to be fused have been processed by Kalman filtering, the quality of the fused image can be improved.

[0126] Please refer to Figure 4 , Figure 4 The structure diagram of the image fusion device disclosed in the embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the image fusion device can include: Figure 4

[0127] The acquisition module 410 is configured to acquire n frames of to-be-processed images, where n is a positive integer.

[0128] The optimization module 420 is configured to calculate the Kalman gain value and the actual pixel value of the K-n+1th to-be-processed image according to the initial prediction deviation and the initial measurement deviation, the prediction pixel value and the measurement pixel value of the K-n+1th to-be-processed image, where K is a positive integer greater than or equal to n; and perform recursive calculation according to the Kalman gain value of the K-n+1th to-be-processed image to obtain the Kalman gain value and the actual pixel value of the K-n+2th to-be-processed image, and iterate until the Kalman gain value and the actual pixel value of the Kth to-be-processed image are obtained.

[0129] The fusion module 430 is configured to perform fusion processing on the n frames of to-be-processed images according to the actual pixel values of the n frames of to-be-processed images to obtain a fused image.

[0130] The above device is implemented. The device acquires n frames of to-be-processed images, calculates the Kalman gain value and the actual pixel value of the K-n+1th to-be-processed image according to the initial prediction deviation and the initial measurement deviation, the prediction pixel value and the measurement pixel value of the K-n+1th to-be-processed image, where n is a positive integer and K is a positive integer greater than or equal to n, then further performs recursive calculation according to the Kalman gain value of the K-n+1th to-be-processed image to obtain the Kalman gain value and the actual pixel value of the K-n+2th to-be-processed image, and iterates until the Kalman gain value and the actual pixel value of the Kth to-be-processed image are obtained, and finally performs fusion processing on the n frames of to-be-processed images according to the actual pixel values of the n frames of to-be-processed images to obtain a fused image. It can be seen that, by implementing the embodiment of the present application, the Kalman gain value of the previous frame of image can be used for continuous recursive and iterative calculation, so that the Kalman gain converges to approach the true value, the accuracy of the actual pixel value of the to-be-processed image is improved, image noise is effectively removed, the interference caused by factors such as light, motion and jitter on the image is reduced, errors are reduced, and the smoothness and quality of the fused image are improved.

[0131] ​In some optional embodiments, the optimization module 420 is configured to calculate the Kalman gain value and the actual pixel value of the K-n+1th image to be processed according to the initial prediction deviation, the initial measurement deviation, the predicted pixel value and the measured pixel value of the K-n+1th image to be processed.

[0132] The covariance of the initial prediction deviation and the initial measurement deviation is calculated as the Kalman gain value of the K-n+1th image to be processed.

[0133] The actual pixel value of the K-n+1th image to be processed is calculated according to the Kalman gain value of the K-n+1th image to be processed, the predicted pixel value and the measured pixel value of the K-n+1th image to be processed.

[0134] In some optional embodiments, the optimization module 420 is configured to recursively calculate the Kalman gain value and the actual pixel value of the K-n+2th image to be processed according to the Kalman gain value of the K-n+1th image to be processed.

[0135] The optimized deviation of the K-n+1th image to be processed is calculated according to the Kalman gain value of the K-n+1th image to be processed and the initial prediction deviation.

[0136] The current prediction deviation of the K-n+2th image to be processed is calculated according to the optimized deviation of the K-n+1th image to be processed.

[0137] The covariance of the current prediction deviation of the K-n+2th image to be processed and the initial measurement deviation is calculated as the Kalman gain value of the K-n+2th image to be processed.

[0138] The actual pixel value of the K-n+2th image to be processed is calculated according to the Kalman gain value of the K-n+2th image to be processed, the predicted pixel value and the measured pixel value of the K-n+2th image to be processed.

[0139] In some optional embodiments, the optimization module 420 is configured to iterate until the Kalman gain value and the actual pixel value of the Kth image to be processed are obtained.

[0140] When K-n+2 is equal to K, the step of performing fusion processing on the n th image to be processed according to the actual pixel value of the n th image to be processed to obtain a fused image is performed.

[0141] When K-n+2 is less than K, the iteration is performed until the Kalman gain value and the actual pixel value of the Kth image to be processed are obtained.

[0142] Through the above embodiment, the Kalman gain value of the previous frame image is used for recursion and iteration, so that the Kalman gain converges to the real value, and the accuracy of the actual pixel value of the to-be-processed image is improved.

[0143] In some optional embodiments, the fusion module 430 is configured to fuse the n frames of to-be-processed images according to the actual pixel values of the n frames of to-be-processed images, and the fusion manner of the fusion module 430 is specifically as follows:

[0144] According to the actual pixel value of the Kth frame of to-be-processed image and the actual pixel value of the K-1th frame of to-be-processed image, the Kth frame of to-be-processed image and the K-1th frame of to-be-processed image are fused, and the fusion is iterated until the K-n+1th frame of to-be-processed image is fused, and a fusion image is obtained.

[0145] In the above embodiment, since the to-be-processed image to be fused is denoised in advance, the pixel quality of each frame of to-be-processed image is improved, so that the fusion quality after fusion is also improved, and the smoothness is better, which is suitable for the application scene of the panoramic stitching algorithm.

[0146] Referring to Figure 5 , Figure 5 a structural schematic diagram of an electronic device disclosed in an embodiment of the present application is shown in the accompanying drawings; Figure 5 The electronic device shown in the accompanying drawings can include:

[0147] a memory 501 storing executable program codes;

[0148] a processor 502 coupled with the memory 501;

[0149] The processor 502 calls the executable program codes stored in the memory 501 to execute Figures 1 to 3 part or all of the steps of any one of the image fusion methods.

[0150] The present application also discloses a computer readable storage medium storing a computer program, wherein the computer program causes a computer to execute Figures 1 to 3 the disclosed image fusion method.

[0151] The present application also discloses a computer program product, when the computer program product runs on a computer, causes the computer to execute Figures 1 to 3 part or all of the steps of any one of the disclosed methods.

[0152] The present application also discloses an application publishing platform, which is used to publish a computer program product, wherein when the computer program product runs on a computer, causes the computer to execute Figures 1 to 3part or all of the steps of any of the methods disclosed.

[0153] Those skilled in the art can understand that all or part of the steps in the above-mentioned various methods of the embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including a Read-Only Memory (ROM), a Random Access Memory (RAM), a Programmable Read-only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), a One-time Programmable Read-Only Memory (OTPROM), an Electrically-Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, a magnetic disk storage, a magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0154] The above describes in detail the image fusion method, device, electronic device and storage medium disclosed in the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples. The above embodiment descriptions are only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. An image fusion method, characterized in that, include: Acquire n frames of images to be processed, where n is a positive integer; Calculate the covariance between the initial prediction bias and the initial measurement bias, and use it as the Kalman gain value of the K-n+1th frame of the image to be processed; and calculate the actual pixel value of the K-n+1th frame of the image to be processed based on the Kalman gain value of the K-n+1th frame of the image to be processed, the predicted pixel value of the K-n+1th frame of the image to be processed, and the measured pixel value of the K-n+1th frame of the image to be processed, where K is a positive integer greater than or equal to n; Based on the Kalman gain value and initial prediction bias of the K-n+1th frame of the image to be processed, the optimized bias of the K-n+1th frame of the image to be processed is calculated; and based on the optimized bias of the K-n+1th frame of the image to be processed, the current prediction bias of the K-n+2th frame of the image to be processed is calculated; and the covariance between the current prediction bias and the initial measurement bias of the K-n+2th frame of the image to be processed is calculated as the Kalman gain value of the K-n+2th frame of the image to be processed; and based on the Kalman gain value, the predicted pixel value, and the measured pixel value of the K-n+2th frame of the image to be processed, the actual pixel value of the K-n+2th frame of the image to be processed is calculated, and this process is iterated until the Kalman gain value and the actual pixel value of the Kth frame of the image to be processed are obtained. The n frames of images to be processed are fused based on their actual pixel values ​​to obtain a fused image.

2. The method according to claim 1, characterized in that, The iteration until the K-th frame of the image to be processed is obtained includes: When K-n+2 equals K, the step of performing fusion processing on the n frames of images to be processed based on the actual pixel values ​​of the n frames of images to be processed to obtain a fused image is executed. When K-n+2 is less than K, iterate until the Kalman gain value and actual pixel value of the Kth frame of the image to be processed are obtained.

3. The method according to any one of claims 1 to 2, characterized in that, The step of fusing the n frames of images to be processed based on their actual pixel values ​​to obtain a fused image includes: Based on the actual pixel values ​​of the Kth frame and the (K-1)th frame to be processed, the Kth frame and the (K-1)th frame to be processed are fused together, and the process is iterated until the (K-n+1)th frame to be processed is completely fused to obtain the fused image.

4. An image fusion apparatus, characterized in that, include: The acquisition module is used to acquire n frames of images to be processed, where n is a positive integer; The optimization module is used to calculate the covariance between the initial prediction bias and the initial measurement bias, as the Kalman gain value of the (K-n+1)th frame of the image to be processed; and to calculate the actual pixel value of the (K-n+1)th frame of the image to be processed based on the Kalman gain value, the predicted pixel value, and the measured pixel value, where K is a positive integer greater than or equal to n; and to calculate the optimization bias of the (K-n+1)th frame of the image to be processed based on the Kalman gain value and the initial prediction bias, and to calculate the actual pixel value of the (K-n+1)th frame of the image to be processed based on the Kalman gain value and the initial measurement bias. The optimization deviation calculation of the n+1 frame of the image to be processed is used to obtain the current prediction deviation of the K-n+2 frame of the image to be processed. The covariance between the current prediction deviation of the K-n+2 frame of the image to be processed and the initial measurement deviation is calculated as the Kalman gain value of the K-n+2 frame of the image to be processed. Based on the Kalman gain value of the K-n+2 frame of the image to be processed, the predicted pixel value and the measured pixel value of the K-n+2 frame of the image to be processed, the actual pixel value of the K-n+2 frame of the image to be processed is calculated, and the process is iterated until the Kalman gain value and the actual pixel value of the K frame of the image to be processed are obtained. The fusion module is used to perform fusion processing on the n frames of images to be processed based on the actual pixel values ​​of the n frames to be processed, so as to obtain a fused image.

5. The apparatus according to claim 4, characterized in that, The optimization module iterates until the Kalman gain value and actual pixel value of the Kth frame of the image to be processed are obtained in the following way: When K-n+2 equals K, the step of performing fusion processing on the n frames of images to be processed based on the actual pixel values ​​of the n frames of images to be processed to obtain a fused image is executed. When K-n+2 is less than K, iterate until the Kalman gain value and actual pixel value of the Kth frame of the image to be processed are obtained.

6. The apparatus according to any one of claims 4 to 5, characterized in that, The fusion module is used to fuse the n frames of images to be processed based on their actual pixel values, and the specific method for obtaining the fused image is as follows: Based on the actual pixel values ​​of the Kth frame and the (K-1)th frame to be processed, the Kth frame and the (K-1)th frame to be processed are fused together, and the process is iterated until the (K-n+1)th frame to be processed is completely fused to obtain the fused image.

7. An electronic device, characterized in that, include: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute an image fusion method according to any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1-3.

Images