Image processing method, image processing device, terminal and storage medium

Abstract

The present application discloses an image processing method, an image processing device, a terminal, and a computer-readable storage medium. The image processing method includes: obtaining multiple frames of original images; selecting one frame from the multiple frames of original images as a reference image, and using the remaining frames as input images; obtaining an inter-frame difference image based on the difference between the input image and the reference image; determining a noise feature weight based on noise threshold information corresponding to the inter-frame difference image and the reference image; determining a local feature weight based on local feature information corresponding to the inter-frame difference image and the reference image; determining a fusion weight based on the noise feature weight and the local feature weight; and fusing multiple frames of original images based on the fusion weight to obtain a fused image. The image processing method, image processing device, terminal, and computer-readable storage medium of the embodiments of the present application can achieve a better multi-frame fusion effect and generate a high-quality fused image with clear images and no ghosting.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an image processing device, a terminal, and a computer-readable storage medium. Background Art

[0002] With the continuous development of image acquisition devices such as mobile phones, charge-coupled device (CCD) cameras, and complementary metal-oxide-semiconductor (CMOS) cameras, people's demand for high-resolution images continues to grow, and their pursuit of a high-quality visual experience is also becoming increasingly higher. To improve imaging quality, the industry has proposed multi-frame fusion technology, which utilizes the complementary information between multiple frames to fuse information, thereby reconstructing details, reducing noise, and improving image clarity.

[0003] However, in practical applications, due to camera shake or objects in relative motion within the scene, multiple frames can overlap during the fusion process, significantly reducing image quality and affecting visual perception. This overlapping and misaligned pixel phenomenon during the fusion process is known as "ghosting" artifacts. Removing ghosting and avoiding image blur has become a pressing challenge for those skilled in the art. Summary of the Invention

[0004] Embodiments of the present application provide an image processing method, an image processing device, a terminal, and a computer-readable storage medium.

[0005] The image processing method according to the embodiment of the present application includes:

[0006] Get multiple frames of original images;

[0007] Selecting one frame from the multiple frames of the original image as a reference image and using the remaining frames as input images;

[0008] obtaining an inter-frame difference image according to the difference between the input image and the reference image;

[0009] determining a noise feature weight according to noise threshold information corresponding to the inter-frame difference image and the reference image;

[0010] Determining a local feature weight according to local feature information corresponding to the inter-frame difference image and the reference image;

[0011] Determining a fusion weight according to the noise feature weight and the local feature weight;

[0012] A fusion process is performed on multiple frames of the original image according to the fusion weight to obtain a fused image.

[0013] The image processing device according to an embodiment of the present application includes:

[0014] An acquisition module is used to acquire multiple frames of original images;

[0015] A selection module, configured to select one frame from the plurality of frames of the original image as a reference image and use the remaining frames as input images;

[0016] a processing module, configured to obtain an inter-frame difference image according to a difference between the input image and the reference image;

[0017] A first determining module, configured to determine a noise feature weight according to noise threshold information corresponding to the inter-frame difference image and the reference image;

[0018] A second determining module is configured to determine a local feature weight based on local feature information corresponding to the inter-frame difference image and the reference image;

[0019] A third determining module, configured to determine a fusion weight according to the noise feature weight and the local feature weight;

[0020] The fusion module is used to fuse the multiple frames of the original image according to the fusion weight to obtain a fused image.

[0021] The terminal according to the embodiment of the present application includes one or more processors and a memory, wherein the memory stores a computer program. When the computer program is executed by the processor, the image processing method according to the embodiment of the present application is implemented.

[0022] The computer-readable storage medium of the embodiment of the present application stores a computer program thereon, and when the program is executed by a processor, the image processing method of the embodiment of the present application is implemented.

[0023] In the image processing method, image processing device, terminal and computer-readable storage medium of the embodiments of the present application, for the selection of fusion weights when fusing multiple frames of original images, on the one hand, the noise feature weights are determined based on the inter-frame difference image and the noise threshold information, and on the other hand, the local feature weights are determined based on the inter-frame difference image and the local feature information. Finally, the fusion weights are determined by combining the noise feature weights and the local feature weights, which can obtain a better multi-frame fusion effect and generate a high-quality fused image with clear images and no ghosting.

[0024] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0026] Figure 1 is a flowchart of an image processing method according to certain embodiments of the present application;

[0027] Figure 2 is a schematic diagram of a module of an image processing device according to certain embodiments of the present application;

[0028] Figure 3 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0029] Figure 4 is a flowchart of an image processing method according to certain embodiments of the present application;

[0030] Figure 5 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0031] Figure 6 is a flowchart of an image processing method according to certain embodiments of the present application;

[0032] Figure 7 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0033] Figure 8 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0034] Figure 9 is a flowchart of an image processing method according to certain embodiments of the present application;

[0035] Figure 10 is a flowchart of an image processing method according to certain embodiments of the present application;

[0036] Figure 11 is a flowchart of an image processing method according to certain embodiments of the present application;

[0037] Figure 12 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0038] Figure 13 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0039] Figure 14 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0040] Figure 15 is a flowchart of an image processing method according to certain embodiments of the present application;

[0041] Figure 16 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0042] Figure 17 is a flowchart of an image processing method according to certain embodiments of the present application;

[0043] Figure 18 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0044] Figure 19 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0045] Figure 20 is a flowchart of an image processing method according to certain embodiments of the present application;

[0046] Figure 21 is a flowchart of an image processing method according to certain embodiments of the present application;

[0047] Figure 22 is a schematic diagram of the principle of the image processing method of certain embodiments of the present application;

[0048] Figure 23 This is a schematic diagram of the effect of the image processing method in some embodiments of the present application;

[0049] Figure 24 is a schematic structural diagram of a terminal in some embodiments of the present application;

[0050] Figure 25 This is a schematic diagram of the connection status between a computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0051] The following further describes the embodiments of the present application in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions. Furthermore, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary and are intended only to explain the embodiments of the present application and are not to be construed as limiting the present application.

[0052] See also Figure 1 , the embodiment of the present application provides an image processing method. The image processing method includes:

[0053] 01: Get multiple frames of original images;

[0054] 02: Select one frame from multiple original images as the reference image, and use the remaining frames as input images;

[0055] 03: Obtain an inter-frame difference image based on the difference between the input image and the reference image;

[0056] 04: Determine the noise feature weight based on the noise threshold information corresponding to the inter-frame difference image and the reference image;

[0057] 05: Determine the local feature weight based on the local feature information corresponding to the inter-frame difference image and the reference image;

[0058] 06: Determine the fusion weight based on the noise feature weight and the local feature weight;

[0059] 07: Fusion processing is performed on multiple frames of original images according to the fusion weight to obtain a fused image.

[0060] See also Figure 2 , the embodiment of the present application also provides an image processing device 100. The image processing device 100 includes an acquisition module 10, a selection module 20, a processing module 30, a first determination module 40, a second determination module 50, a third determination module 60, and a fusion module 70. The image processing method of the embodiment of the present application can be implemented by the image processing device 100 of the embodiment of the present application. Specifically, the acquisition module 10 can be used to implement the method in 01, the selection module 20 can be used to implement the method in 02, the processing module 30 can be used to implement the method in 03, the first determination module 40 can be used to implement the method in 04, the second determination module 50 can be used to implement the method in 05, the third determination module 60 can be used to implement the method in 06, and the fusion module 70 can be used to implement the method in 07. In other words, the acquisition module 10 can be used to acquire multiple frames of original images. The selection module 20 can be used to select one frame from the multiple frames of original images as a reference image and the remaining frames as input images. The processing module 30 can be used to obtain an inter-frame difference image based on the difference between the input image and the reference image. The first determination module 40 can be configured to determine a noise feature weight based on noise threshold information corresponding to the inter-frame difference image and the reference image. The second determination module 50 can be configured to determine a local feature weight based on local feature information corresponding to the inter-frame difference image and the reference image. The third determination module 60 can be configured to determine a fusion weight based on the noise feature weight and the local feature weight. The fusion module 70 can be configured to fuse multiple original image frames based on the fusion weight to generate a fused image.

[0061] In the image processing method and image processing device 100 of the embodiment of the present application, for the selection of fusion weights when fusing multiple frames of original images, on the one hand, the noise feature weights are determined based on the inter-frame difference image and the noise threshold information, and on the other hand, the local feature weights are determined based on the inter-frame difference image and the local feature information. Finally, the fusion weights are determined by combining the noise feature weights and the local feature weights. This can achieve a better multi-frame fusion effect and generate a high-quality fused image with clear images and no ghosting.

[0062] Specifically, the multiple frames of original images are at least two frames of original images. Figure 3 , taking the multiple frames of original images as N frames of original images as an example, N ≥ 2, N is an integer. At this time, the image processing method process is as follows:

[0063] Get N frames of original images;

[0064] Select one frame from N original images as the reference image, and use the remaining N-1 frames as input images;

[0065] Obtain N-1 frame inter-frame difference images based on the difference between the N-1 frame input image and the reference image;

[0066] Determine the noise feature weight of the N-1 frame according to the noise threshold information corresponding to the N-1 frame inter-frame difference image and the reference image;

[0067] Determine the local feature weight of the N-1 frame according to the local feature information corresponding to the N-1 frame inter-frame difference image and the reference image;

[0068] Determine the N-1 frame fusion weight according to the N-1 frame noise feature weight and the N-1 frame local feature weight;

[0069] According to the N-1 frame fusion weight, the N-1 frame input image is fused with the reference image to obtain a fused image frame.

[0070] That is to say, for each frame of input image, it is necessary to calculate the difference between it and the reference image to obtain the inter-frame difference image, so as to determine the corresponding noise feature weight and local feature weight, and then obtain the fusion weight corresponding to the frame of input image. Finally, each frame of input image is fused with the reference image using the corresponding fusion weight, and finally a frame of fused image is output.

[0071] For example, when N = 2, two frames of original images are obtained. Taking P0 as the reference image and P1 as the input image from the original image as an example, the inter-frame difference image ΔP1 is obtained based on the difference between the input image P1 and the reference image P0. Then, on the one hand, the noise feature weight W1 is determined based on the noise threshold information corresponding to the inter-frame difference image ΔP1 and the reference image P0. On the other hand, the local feature weight R1 is determined based on the local feature information corresponding to the inter-frame difference image ΔP1 and the reference image P0. The fusion weight F1 is then determined based on the noise feature weight W1 and the local feature weight R1. Finally, the input image P1 and the reference image P0 are fused based on the fusion weight F1 to obtain the fused image P.

[0072] For another example, when N=3, 3 frames of original images are acquired. Taking P0 as the reference image and P1 and P2 as the input images from the original image as an example, the inter-frame difference image ΔP1 is obtained according to the difference between the input image P1 and the reference image P0, and the inter-frame difference image ΔP2 is obtained according to the difference between the input image P2 and the reference image P0; then, on the one hand, the noise feature weight W1 is determined according to the noise threshold information corresponding to the inter-frame difference image ΔP1 and the reference image P0, and the noise feature weight W2 is determined according to the noise threshold information corresponding to the inter-frame difference image ΔP2 and the reference image P0; on the other hand, the local feature weight R1 is determined according to the local feature information corresponding to the inter-frame difference image ΔP1 and the reference image P0, and the local feature weight R2 is determined according to the local feature information corresponding to the inter-frame difference image ΔP2 and the reference image P0; then, the fusion weight F1 is determined according to the noise feature weight W1 and the local feature weight R1, and the fusion weight F2 is determined according to the noise feature weight W2 and the local feature weight R2; finally, the input image P1 is fused according to the fusion weight F1, and the input image P2 is fused with the reference image P0 according to the fusion weight F2 to obtain the fused image P.

[0073] In the implementation mode of the present application, in order to obtain a better multi-frame fusion effect and avoid the generation of ghosting, a unified photographing strategy should be set as much as possible when acquiring multiple frames of original images (that is, the same photographing parameters should be used). For example, to ensure that the image brightness of multiple frames of original images is consistent, the exposure time and shutter time are kept consistent when acquiring multiple frames of original images. Among them, the exposure value EV = 0, the operating frequency of the image sensor is 24fps, and the output time of each frame is 40ms. In addition, functional algorithms such as anti-shake and focus can be added to ensure that the original image is clear and that there is no obvious deformation such as scaling between frames, so as not to affect the multi-frame image fusion effect.

[0074] See also Figure 4 In some embodiments, selecting a frame from multiple frames of original images as a reference image includes:

[0075] 021: Get the image brightness information of each frame of the original image;

[0076] 022: Calculate the image brightness mean of each frame of the original image based on the image brightness information;

[0077] 023: taking the frame with the largest average brightness value among the multiple frames of original images as the reference image; and / or

[0078] 024: Get the image gradient information of each frame of the original image;

[0079] 025: Calculate the image gradient mean of each frame of the original image based on the image gradient information;

[0080] 026: The frame with the largest image gradient mean among multiple frames of original images is used as the reference image.

[0081] See also Figure 2 In some embodiments, the selection module 20 can be used to implement the methods in 021, 022, 023, 024, 025, and 026. That is, the selection module 20 can be used to: obtain image brightness information of each frame of the original image; calculate the image brightness mean of each frame of the original image based on the image brightness information; use the frame with the largest image brightness mean among the multiple frames of original image as the reference image; and / or obtain image gradient information of each frame of the original image; calculate the image gradient mean of each frame of the original image based on the image gradient information; and use the frame with the largest image gradient mean among the multiple frames of original image as the reference image.

[0082] Specifically, the quality of the reference image affects the final image fusion effect to a certain extent. In order to obtain a fused image of better quality, the reference image should be as bright and clear as possible and rich in details. Therefore, the embodiment of the present application selects the reference image from the two perspectives of image brightness and image gradient, and selects the original image with the largest image brightness mean and the original image with the largest image gradient mean among the multiple frames of original images as the reference image. A larger image brightness mean can ensure that the reference image has clear brightness, and a larger image gradient mean can ensure that the reference image has rich details.

[0083] See also Figure 5 Each frame of the original image includes multiple pixels. Obtaining the image brightness information of each frame of the original image is to obtain the brightness values ​​of multiple pixels. The image brightness mean is to calculate the brightness mean based on the brightness values ​​of multiple pixels. The calculation formula of the image brightness mean can be shown as follows:

[0084] L=∑I(x,y) / (h*w)((0,0)<(x,y)<(h,w))

[0085] Where L is the mean brightness of the image, I is the brightness value of the pixel, x and y are the rows and columns of the pixel, respectively, and h and w are the height (total number of rows of pixels) and width (total number of columns of pixels) of the original image, respectively.

[0086] Obtaining the image gradient information of each frame of the original image is to obtain the gradient values ​​of multiple pixels. The image gradient mean is to calculate the gradient mean based on the gradient values ​​of multiple pixels. The calculation formula of the image gradient mean can be shown as follows:

[0087] dx = I(x, y) – I(x-1, y)

[0088] dy = I(x, y) – I(x, y - 1)

[0089] d=sqrt(dx^2+dy^2)

[0090] D = ∑d(x, y) / (h*w)

[0091] Where D is the mean image gradient, d is the gradient value of the pixel, dx is the row gradient, dy is the column gradient, I is the brightness value of the pixel, x and y are the row and column of the pixel respectively, h and w are the height (total number of rows of pixels) and width (total number of columns of pixels) of the original image respectively.

[0092] by Figure 5 For example, each frame of the original image consists of 4*4 pixels (this is just a simple illustration; the number of pixels in reality is much larger). The brightness values ​​of these 16 pixels are: A11, A12, A13, A14, A21, A22, A23, A24, A31, A32, A33, A34, A41, A42, A43, and A44. The average image brightness L = (A11+A12+A13+A14+A21+A22+A23+A24+A31+A32+A33+A34+A41+A42+A43+A44) / 4*4. The mean image gradient is calculated as follows: Taking pixel 32 as an example, the row gradient dx = A32 - A22, the column gradient dy = A32 - A31, and the pixel gradient value d = sqrt((A32 - A22)^2 + (A32 - A31)^2). Similarly, the gradient values ​​of the 16 pixels can be calculated as follows: d11, d12, d13, d14, d21, d22, d23, d24, d31, d32, d33, d34, d41, d42, d43, d44. The mean image gradient D = (d11 + d12 + d13 + d14 + d21 + d22 + d23 + d24 + d31 + d32 + d33 + d34 + d41 + d42 + d43 + d44) / 4 * 4.

[0093] It should be noted that the method of obtaining the image gradient information of each frame of the original image and calculating the image gradient mean based on the image gradient information is not limited to this. In other examples, some image gradient algorithms (such as the Sobel operator, the Laplacian operator, the Prewitt operator, the Roberts operator, the Canny operator, etc.) can also be used, which is not limited here.

[0094] After obtaining the image brightness mean of each frame of the original image, the frame with the largest image brightness mean among the multiple frames of the original image is selected as the reference image; and / or after obtaining the image gradient mean of each frame of the original image, the frame with the largest image gradient mean among the multiple frames of the original image is selected as the reference image. In other words, when selecting the reference image, the image brightness mean can be used alone to ensure that the reference image has clear brightness; the image gradient mean can be used alone to ensure that the reference image has rich details; or both the image brightness mean and the image gradient mean can be used to ensure that the reference image has clear brightness and rich details. In the last case, the frame with the largest image brightness mean and the largest image gradient mean among the multiple frames of the original image can be selected as the reference image. Alternatively, when there are multiple frames of the original image with the largest image brightness mean, the frame with the larger image gradient mean can be further selected as the reference image. Alternatively, when there are multiple frames of the original image with the largest image gradient mean, the frame with the larger image brightness mean can be further selected as the reference image.

[0095] See also Figure 6 In some embodiments, obtaining an inter-frame difference image (i.e., 03) based on the difference between the input image and the reference image includes:

[0096] 031: Calculate the difference between the input image and the reference image to obtain an initial difference image;

[0097] 032: Smoothing the initial difference image to obtain an inter-frame difference image.

[0098] See also Figure 2 In some embodiments, the processing module 30 can be used to implement the methods in 031 and 032. That is, the processing module 30 can be used to: calculate the difference between the input image and the reference image to obtain an initial difference image; and perform smoothing on the initial difference image to obtain an inter-frame difference image.

[0099] Specifically, when calculating the difference between the input image and the reference image, the brightness values ​​of the input image and the reference image at corresponding pixel positions can be subtracted to obtain the initial difference image. The specific calculation formula can be as follows:

[0100] di(x,y)=abs(I(x,y)-Base(x,y))

[0101] Where di is the brightness difference of the pixel in the initial difference image, I is the brightness value of the pixel in the input image, Base is the brightness value of the pixel in the reference image, and x and y are the row and column of the pixel respectively.

[0102] by Figure 7For example, if the brightness value of pixel 11 in the input image is B11 and the brightness value of pixel 11 in the reference image is C11, then the brightness difference di11 of pixel 11 in the initial difference image is |B11-C11|. If the brightness value of pixel 12 in the input image is B12 and the brightness value of pixel 12 in the reference image is C12, then the brightness difference di12 of pixel 12 in the initial difference image is |B12-C12|. Similarly, the brightness difference di of each pixel in the initial difference image can be calculated to obtain a frame of initial difference image.

[0103] See also Figure 8 In order to reduce the error caused by singular points, after obtaining the initial difference image, the first sampling window (which can be set according to empirical values, such as a 3×3 sampling window) can be used to smooth the initial difference image to obtain the inter-frame difference image. The specific calculation formula is as follows:

[0104]

[0105] Where d is the difference value of the pixel in the inter-frame difference image (i.e., the brightness difference after smoothing), di is the brightness difference of the pixel in the initial difference image, kernel is a 3×3 Gaussian convolution kernel, and x and y are the rows and columns of the pixel, respectively.

[0106] And so on, such as Figure 3 As shown, when there are N-1 frames of input images, each frame of the input image can be subtracted from the reference image and then smoothed to obtain N-1 frames of inter-frame difference images, which will not be explained here one by one.

[0107] See also Figure 9 In some embodiments, determining the noise feature weight (i.e., 04) based on the noise threshold information corresponding to the inter-frame difference image and the reference image includes:

[0108] 041: Estimating the noise threshold of the reference image to obtain noise threshold information;

[0109] 042: Performing a first motion detection based on a magnitude relationship between the inter-frame difference image and the noise threshold information to determine an initial noise weight;

[0110] 043: Perform morphological processing on the initial noise weight to obtain the noise feature weight.

[0111] See also Figure 2In some embodiments, the first determination module 40 may be used to implement the methods in 041, 042, and 043. That is, the first determination module 40 may be used to: perform noise threshold estimation on the reference image to obtain noise threshold information; perform first motion detection based on the magnitude relationship between the inter-frame difference image and the noise threshold information to determine an initial noise weight; and perform morphological processing on the initial noise weight to obtain a noise feature weight.

[0112] Specifically, during the first motion detection, the present embodiment uses noise threshold information as the threshold for comparing inter-frame difference images. This approach takes into account the physical mechanisms of camera imaging and the principles of motion imaging, resulting in both principled and adaptive performance. Therefore, it is possible to distinguish between inter-frame differences caused by displacement changes due to the target's motion in the scene and differences caused by physical noise, thereby accurately determining whether motion exists between images and ultimately obtaining accurate noise feature weights.

[0113] Similarly, when there are N-1 frames of input images, the inter-frame difference image corresponding to each frame of input image can be compared with the noise threshold information to perform the first motion detection, thereby obtaining the N-1 frame initial noise weights, and then the initial noise weights of each frame are morphologically processed to obtain the N-1 frame noise feature weights, which will not be explained one by one here.

[0114] See also Figure 10 In some embodiments, performing noise threshold estimation on a reference image to obtain noise threshold information (i.e., 041) includes:

[0115] 0411: Get the original noise threshold based on the sensitivity when the original image was captured;

[0116] 0412: Obtain image brightness information of the reference image;

[0117] 0413: Determine the noise threshold information based on the image brightness information and the original noise threshold.

[0118] See also Figure 2 In some embodiments, the first determination module 40 may be used to implement the methods in 0411, 0412, and 0413. That is, the first determination module 40 may be used to: obtain an original noise threshold based on the light sensitivity at which the original image was captured; obtain image brightness information of a reference image; and determine noise threshold information based on the image brightness information and the original noise threshold.

[0119] Specifically, as mentioned above, in order to obtain a better multi-frame fusion effect and avoid the generation of ghosting, the same shooting parameters are used when acquiring multiple frames of raw images. Therefore, the sensitivity when shooting multiple frames of raw images is the same, and a unified sensitivity can be used here. In addition, compared with the YUV domain, the RAW domain noise, especially the Raw image noise directly output by the image sensor, is simpler in form. Therefore, the image types involved in determining the noise threshold information can all use Raw images. The Gaussian-Poisson mathematical model can also be used to calibrate the RAW domain noise to conveniently and accurately judge inter-frame motion.

[0120] In one example, the original noise threshold includes a Poisson noise parameter and a Gaussian noise parameter. The formula for obtaining the original noise threshold based on the sensitivity when the original image was captured is as follows:

[0121] a=-0.83x+2.97

[0122] b=1.73x–4.24

[0123] Among them, a is the Poisson noise parameter, b is the Gaussian noise parameter, and x is the sensitivity when the original image was taken.

[0124] The reference image includes multiple pixels. Obtaining the image brightness information of the reference image is to obtain the brightness values ​​of the multiple pixels. Determining the noise threshold information based on the image brightness information and the original noise threshold is to calculate the noise threshold of each pixel based on the brightness value of each pixel in the reference image and the original noise threshold. The specific calculation formula can be as follows:

[0125]

[0126] Among them, std is the noise threshold, y x is the brightness value of the pixel, a is the Poisson noise parameter, and b is the Gaussian noise parameter.

[0127] by Figure 7 For example, the reference image consists of 4*4 pixels, and the brightness values ​​of these 16 pixels are: C11, C12, C13, C14, C21, C22, C23, C24, C31, C32, C33, C34, C41, C42, C43, C44. Then the noise threshold of pixel 11 in the reference image is Noise threshold for pixel 12 in the reference image Noise threshold for pixel 13 in the reference image By analogy, the noise threshold std of each pixel in the reference image can be calculated to obtain a frame of noise threshold information (such as Figure 12 shown).

[0128] The embodiments of this application set noise thresholds (std) for different pixel regions, resulting in better adaptability. This avoids the weaker adaptability that can occur when directly setting a global threshold parameter without distinguishing between different brightness regions (e.g., dark and bright areas). It is understood that when directly setting a global threshold parameter, if the threshold is set too low, most regions will be processed as a single frame, failing to increase image detail. If the threshold is set too high, even slight motion will be interpreted as a stationary region and fused, resulting in ghosting and affecting the multi-frame fusion effect.

[0129] See also Figure 11 In some embodiments, performing a first motion detection based on a magnitude relationship between an inter-frame difference image and noise threshold information to determine an initial noise weight (i.e., 042) includes:

[0130] 0421: Compare the size of the inter-frame difference image and the noise threshold information at the corresponding pixel positions;

[0131] 0422: When the difference value of the inter-frame difference image at the corresponding pixel position is greater than the noise threshold value of the noise threshold information at the corresponding pixel position, the area where the corresponding pixel position is located in the input image corresponding to the inter-frame difference image is marked as a motion area;

[0132] 0423: When the difference value of the inter-frame difference image at the corresponding pixel position is less than the noise threshold value of the noise threshold information at the corresponding pixel position, the area where the corresponding pixel position is located in the input image corresponding to the inter-frame difference image is marked as a static area;

[0133] 0424: The weight value of the moving area is set to the first initial noise weight, and the weight value of the stationary area is set to the second initial noise weight; wherein the first initial noise weight is smaller than the second initial noise weight.

[0134] See also Figure 2 In some embodiments, the first determination module 40 may be used to implement the methods in 0421, 0422, 0423, and 0424. That is, the first determination module 40 may be used to: compare the size of the inter-frame difference image and the noise threshold information at the corresponding pixel position; when the difference value of the inter-frame difference image at the corresponding pixel position is greater than the noise threshold value of the noise threshold information at the corresponding pixel position, mark the region of the input image corresponding to the inter-frame difference image at the corresponding pixel position as a motion region; when the difference value of the inter-frame difference image at the corresponding pixel position is less than the noise threshold value of the noise threshold information at the corresponding pixel position, mark the region of the input image corresponding to the inter-frame difference image at the corresponding pixel position as a static region; set the weight value of the motion region to a first initial noise weight, and set the weight value of the static region to a second initial noise weight; wherein the first initial noise weight is less than the second initial noise weight.

[0135] Specifically, Figure 12 For example, the size of the inter-frame difference image and the noise threshold information at the corresponding pixel position is compared. For example, when the difference value d11 of pixel 11 in the inter-frame difference image is greater than the noise threshold std11 of pixel 11 in the noise threshold information, the area where pixel 11 in the input image corresponding to the inter-frame difference image is located is marked as a motion area ( Figure 12 (shown as shaded in the middle); for another example, when the difference value d12 of pixel 12 in the inter-frame difference image is less than the noise threshold value std12 of pixel 12 in the noise threshold information, the region of the input image corresponding to pixel 12 in the inter-frame difference image is marked as a static region. When the difference value d at the corresponding pixel position in the inter-frame difference image is equal to the noise threshold value std at the corresponding pixel position in the noise threshold information, the region of the input image corresponding to the pixel in the inter-frame difference image can be marked as either a moving region or a static region, without limitation. Similarly, all moving and static regions in the input image can be marked.

[0136] Furthermore, the weight value of the motion area can be set to a first initial noise weight, and the weight value of the static area can be set to a second initial noise weight. The first initial noise weight is smaller than the second initial noise weight. For example, the first initial noise weight is 0, and the second initial noise weight is 1 (in other examples, the initial noise weight can also be set to more stages of weights, for example, it can be set to a linear decreasing function according to the situation of the first motion detection. When the size difference between the inter-frame difference image and the noise threshold information at the corresponding pixel position is greater, the initial noise weight is set to a smaller value; when the size difference between the inter-frame difference image and the noise threshold information at the corresponding pixel position is smaller, the initial noise weight is set to a larger value).

[0137] It can be understood that when the difference value is greater than the noise threshold, it means that the difference between frames is mainly caused by motion, and the corresponding area is marked as a motion area. The fusion of this area will produce ghosting, so the weight value can be set to a lower value; and when the difference value is less than the noise threshold, it means that the difference between frames is mainly caused by random noise, and there is no motion between frames, and the corresponding area is marked as a static area. This area can be directly fused, so the weight value can be set to a higher value. In this way, the embodiment of the present application can perform differentiated processing on different brightness areas in the same frame input image and set different initial noise weights, which is conducive to improving the clarity of the fused image and reducing the generation of ghosting. For example, it can reduce the motion misjudgment caused by noise in dark and flat areas, effectively retain image details, and allow users to obtain better quality high-definition images.

[0138] Similarly, when there are N-1 frames of input images, the inter-frame difference image corresponding to each frame of input image can be compared with the noise threshold information to determine the size of the corresponding pixel positions, so as to mark the moving area and the static area in the corresponding input image, and set the first initial noise weight and the second initial noise weight accordingly, thereby obtaining the N-1 frame initial noise weight, which will not be explained in detail here.

[0139] See also Figure 13 After obtaining the initial noise weight, the initial noise weight can be subjected to morphological processing to obtain the noise feature weight. The morphological processing may include corrosion processing and / or expansion processing. Figure 13 and Figure 14 As shown in the figure, assuming that the original image contains moving cars, for example, the white areas in the initial noise weights are multiple moving cars. The initial noise weights only show the outlines of the cars, and the area between the cars is disconnected, giving a sense of fragmentation. After morphological processing, i.e., corrosion and dilation processing, the car area forms a connected domain, which is reflected in the pixel image as follows Figure 13 As shown, the area with a weight value of 0 surrounded by the area with a weight value of 1 is changed to a weight value of 1, thereby obtaining a noise feature weight corresponding to the initial noise weight. In the embodiment of the present application, the noise feature weight is obtained by connecting the binary initial noise weights, which can avoid the sense of fragmentation in the final fused image due to the discrete weights, making the fused image smoother and the fusion effect better.

[0140] See also Figure 15 In some embodiments, determining a local feature weight (i.e., 05) based on local feature information corresponding to an inter-frame difference image and a reference image includes:

[0141] 051: Calculate the local standard deviation of the reference image to obtain local feature information;

[0142] 052: Perform second motion detection based on the difference value of the inter-frame difference image and the local standard deviation of the local feature information;

[0143] 053: Determine the local feature weight according to the result of the second motion detection.

[0144] See also Figure 2 In some embodiments, the second determination module 50 can be used to implement the methods in 051, 052, and 053. That is, the second determination module 50 can be used to: calculate the local standard deviation of the reference image to obtain local feature information; perform a second motion detection based on the difference value of the inter-frame difference image and the local standard deviation of the local feature information; and determine the local feature weight based on the result of the second motion detection.

[0145] Specifically, when performing the second motion detection, the embodiment of the present application combines local feature information on the basis of the inter-frame difference image to accurately judge the inter-frame motion situation, thereby obtaining accurate local feature weights.

[0146] Please combine Figure 16 When the local standard deviation σ of the local feature information is large, it means that the corresponding area of ​​the input image is noisy or belongs to the image edge; when the difference value d of the inter-frame difference image is large, it means that there is obvious motion between frames and the input image is not aligned with the reference image. When the difference value d of the inter-frame difference image is large and the local standard deviation σ of the local feature information is small, it means that the corresponding area of ​​the input image is flat, but the input image is not aligned with the reference image; when the difference value d of the inter-frame difference image is small and the local standard deviation σ of the local feature information is large, it means that there is no obvious motion between frames and the corresponding area of ​​the input image is a noise area, and image fusion is required; when the difference value d of the inter-frame difference image is small and the local standard deviation σ of the local feature information is also small, it means that there is no obvious motion between frames and the corresponding area of ​​the input image is flat. At this time, it can be determined whether to perform image fusion as needed; when the difference value d of the inter-frame difference image is large and the local standard deviation σ of the local feature information is also large, it means that the corresponding area of ​​the input image is an aliasing area and should be discarded.

[0147] In the above cases, if the local standard deviation σ of the local feature information is large, it is preferable to set the local feature weight to a larger value to fuse more image information for noise reduction. If the difference value d of the inter-frame difference image is large, it is preferable to set the local feature weight to a smaller value to avoid ghosting. In other cases, it is necessary to comprehensively consider the size of the local standard deviation σ of the local feature information and the difference value d of the inter-frame difference image, and balance the impact of the two on the image fusion effect to assign appropriate local feature weights.

[0148] Similarly, when there are N-1 frames of input images, the inter-frame difference image corresponding to each frame of input image can be combined with local feature information to perform second motion detection to obtain N-1 frame local feature weights, which will not be explained in detail here.

[0149] See also Figure 17 In some embodiments, determining a local feature weight (i.e., 05) based on local feature information corresponding to an inter-frame difference image and a reference image includes:

[0150] 054: Calculate the local standard deviation of the reference image to obtain local feature information;

[0151] 055: Construct a feature weight relationship between the difference value of the inter-frame difference image, the local standard deviation of the local feature information, and the local feature weight; wherein, when the local standard deviation of the local feature information is constant, the difference value of the inter-frame difference image is negatively correlated with the local feature weight; when the difference value of the inter-frame difference image is constant, the local standard deviation of the local feature information is positively correlated with the local feature weight;

[0152] 056: Determine the local feature weight according to the difference value of the inter-frame difference image, the local standard deviation of the local feature information, and the feature weight relationship.

[0153] See also Figure 2 In some embodiments, the second determination module 50 can be used to implement the methods in 054, 055, and 056. That is, the second determination module 50 can be used to: calculate the local standard deviation of the reference image to obtain local feature information; establish a feature weight relationship between the difference value of the inter-frame difference image, the local standard deviation of the local feature information, and the local feature weight; wherein, when the local standard deviation of the local feature information is constant, the difference value of the inter-frame difference image is negatively correlated with the local feature weight; when the difference value of the inter-frame difference image is constant, the local standard deviation of the local feature information is positively correlated with the local feature weight; and determine the local feature weight based on the difference value of the inter-frame difference image, the local standard deviation of the local feature information, and the feature weight relationship.

[0154] by Figure 18 For example, the reference image consists of 8*8 pixels (this is just a simple illustration; the number of pixels in reality is much larger). First, a second sampling window is selected for each pixel in the reference image (which can be set based on empirical values, such as a 5×5 sampling window). Then, the local standard deviation corresponding to the pixel is calculated using the second sampling window. The specific calculation formula can be as follows:

[0155]

[0156] Where σ is the local standard deviation, I is the brightness value of the pixel in the reference image, μ is the brightness mean of the 5×5 sampling window, and x and y are the rows and columns of the pixel, respectively.

[0157] The local standard deviation σ corresponding to each pixel in the reference image can be calculated in the above manner. By pre-calibration, the feature weight relationship between the difference value d of the inter-frame difference image, the local standard deviation σ of the local feature information, and the local feature weight R can be constructed. This feature weight relationship can be a function model. Taking into account the nonlinear relationship between the above parameters, the embodiment of the present application can select a negative exponential function to construct a ghost removal model (i.e., a feature weight relationship), which can be specifically shown as follows:

[0158]

[0159] Where R is the local feature weight, d is the difference value of the inter-frame difference image, σ is the local standard deviation, α and β are preset parameters, and eps is the minimum value to prevent division by 0 (for example, eps = 0.0001 in this algorithm).

[0160] When the local standard deviation σ of the local feature information is constant, the difference value d of the inter-frame difference image is negatively correlated with the local feature weight R; when the difference value d of the inter-frame difference image is constant, the local standard deviation σ of the local feature information is positively correlated with the local feature weight R. It can be understood that when the inter-frame images are not aligned, the difference value d of the inter-frame difference image is large, so the local feature weight R should be as small as possible to avoid ghosting. When the difference value d of the inter-frame difference image is infinite, the local feature weight R approaches 0. When the difference value d of the inter-frame difference image is small, it indicates that the corresponding regions are likely to be aligned, and more image information is required for fusion, so the local feature weight R can be set larger. When the noise in the flat area is large, the local standard deviation σ of the local feature information is large, and more image information needs to be fused for noise reduction, so the local feature weight R can be set larger.

[0161] After constructing the feature weight relationship between the difference value d of the inter-frame difference image, the local standard deviation σ of the local feature information, and the local feature weight R, the difference value d of the inter-frame difference image and the local standard deviation σ of the local feature information calculated in real time are substituted into the feature weight relationship to obtain the local feature weight R. Figure 19 For example, the difference value d11 of the inter-frame difference image calculated in real time and the local standard deviation σ11 of the local feature information are substituted into the feature weight relationship to obtain the local feature weight R11. The difference value d12 of the inter-frame difference image calculated in real time and the local standard deviation σ12 of the local feature information are substituted into the feature weight relationship to obtain the local feature weight R12. Similarly, the corresponding local feature weight R can be calculated for each pixel in the input image.

[0162] Similarly, when there are N-1 frames of input images, for each frame of input image corresponding to the inter-frame difference image, the local feature weight can be determined in combination with the local feature information to obtain the N-1 frame local feature weight, which will not be explained one by one here.

[0163] See also Figure 20In some embodiments, the inter-frame difference image is obtained by smoothing the initial difference image through a first sampling window. The local feature information is obtained by calculating the local standard deviation of the second sampling window of the reference image. The range of the first sampling window is smaller than the range of the second sampling window. The inter-frame difference image (i.e., 03) is obtained based on the difference between the input image and the reference image, and further includes:

[0164] 033: Perform calibration processing on the inter-frame difference image so that the inter-frame difference image corresponding to the first sampling window matches the local feature information corresponding to the second sampling window.

[0165] See also Figure 2 In some embodiments, the inter-frame difference image is obtained by smoothing the initial difference image through the first sampling window. The local feature information is obtained by calculating the local standard deviation of the second sampling window of the reference image. The range of the first sampling window is smaller than the range of the second sampling window. The processing module 30 can be used to implement the method in 033. In other words, the processing module 30 can be used to calibrate the inter-frame difference image so that the inter-frame difference image corresponding to the first sampling window matches the local feature information corresponding to the second sampling window.

[0166] Specifically, please combine Figure 8 and Figure 18 Considering that the local characteristics and overall noise characteristics of the first sampling window (3×3 sampling window) and the second sampling window (5×5 sampling window) may be different, it is necessary to calibrate the inter-frame difference image to ensure the subsequent matching between the noise feature weight and the local feature weight. By performing parameter analysis simulation (such as Monte Carlo simulation) on a series of images with different brightness levels, the empirical value of the frame difference d can be obtained. md =0.01, and then mathematical modeling can be used to obtain the difference value of the calibrated inter-frame difference image, as shown below:

[0167]

[0168] Among them, d is the difference value of the calibrated inter-frame difference image, d md is the reference value (i.e., the frame difference experience value), d ms is the difference value of the inter-frame difference image obtained by smoothing the first sampling window.

[0169] Similarly, when there are N-1 frames of input images, the inter-frame difference images corresponding to each frame of input image can be calibrated so that the inter-frame difference images corresponding to the first sampling window match the local feature information corresponding to the second sampling window. This will not be explained in detail here.

[0170] See also Figure 21 In some embodiments, determining the fusion weight (i.e., 06) based on the noise feature weight and the local feature weight includes:

[0171] 061: The product of the noise feature weight and the local feature weight is used as the fusion weight.

[0172] See also Figure 2 In some embodiments, the third determining module 60 may be used to implement the method in 061. That is, the third determining module 60 may be used to take the product of the noise feature weight and the local feature weight as the fusion weight.

[0173] Specifically, the calculation formula of the fusion weight can be shown as follows:

[0174] w(i,j)=Weight noise *Weight local

[0175] Among them, w(i,j) is the fusion weight, Weight noise is the noise feature weight, Weight local is the local feature weight, i and j are the rows and columns of pixels respectively.

[0176] by Figure 22 For example ( Figure 22 Only the 4*4 noise feature weight W and the corresponding 4*4 local feature weight R are schematically shown. The fusion weight F11 is the product of the noise feature weight W11 and the local feature weight R11; the fusion weight F12 is the product of the noise feature weight W12 and the local feature weight R12; and the fusion weight F13 is the product of the noise feature weight W13 and the local feature weight R14. Similarly, the fusion weight F corresponding to each pixel in the input image can be calculated.

[0177] Similarly, when there are N-1 frames of input images, for each frame of input image, the product of the noise feature weight and the local feature weight can be used as the fusion weight to obtain the N-1 frame fusion weight, which will not be explained one by one here.

[0178] See also Figure 23 , Figure 23 This figure compares the effects of a fused image obtained using the image processing method of this application with a fused image obtained using a conventional multi-frame fusion algorithm. The figure shows that the fused image obtained using the image processing method of this application is clearer, has more image detail, and achieves a better visual effect. By introducing noise estimation optimization weights, image detail is better preserved in moving areas, resulting in richer and clearer textures around the license plate and tire.

[0179] See also Figure 24The present application also provides a terminal 200. The terminal 200 includes one or more processors 210 and a memory 220. The memory 220 stores a computer program, and when the computer program is executed by the processor 210, the image processing method of any of the above embodiments is implemented.

[0180] For example, when the computer program is executed by the processor 210, the following image processing method is implemented:

[0181] 01: Get multiple frames of original images;

[0182] 02: Select one frame from multiple original images as the reference image, and use the remaining frames as input images;

[0183] 03: Obtain an inter-frame difference image based on the difference between the input image and the reference image;

[0184] 04: Determine the noise feature weight based on the noise threshold information corresponding to the inter-frame difference image and the reference image;

[0185] 05: Determine the local feature weight based on the local feature information corresponding to the inter-frame difference image and the reference image;

[0186] 06: Determine the fusion weight based on the noise feature weight and the local feature weight;

[0187] 07: Fusion processing is performed on multiple frames of original images according to the fusion weight to obtain a fused image.

[0188] For another example, when the computer program is executed by the processor 210, the following image processing method is implemented:

[0189] 021: Get the image brightness information of each frame of the original image;

[0190] 022: Calculate the image brightness mean of each frame of the original image based on the image brightness information;

[0191] 023: taking the frame with the largest average brightness value among the multiple frames of original images as the reference image; and / or

[0192] 024: Get the image gradient information of each frame of the original image;

[0193] 025: Calculate the image gradient mean of each frame of the original image based on the image gradient information;

[0194] 026: The frame with the largest image gradient mean among multiple frames of original images is used as the reference image.

[0195] The terminal 200 in the embodiment of the present application includes but is not limited to a mobile phone, a tablet computer, a camera, a video camera, a personal digital assistant, a wearable device, an intelligent robot, an intelligent vehicle, etc. Among them, the wearable device includes a smart bracelet, a smart watch, smart glasses, etc.

[0196] It should be noted that the explanations of the image processing method and the image processing device 100 in the aforementioned embodiment are also applicable to the terminal 200 in the embodiment of the present application and will not be elaborated here.

[0197] See also Figure 25 The present application also provides a computer-readable storage medium 300 on which a computer program 310 is stored. When the program 310 is executed by the processor 320, the image processing method of any of the above embodiments is implemented.

[0198] For example, when the program 310 is executed by the processor 320, the following image processing method is implemented:

[0199] 01: Get multiple frames of original images;

[0200] 02: Select one frame from multiple original images as the reference image, and use the remaining frames as input images;

[0201] 03: Obtain an inter-frame difference image based on the difference between the input image and the reference image;

[0202] 04: Determine the noise feature weight based on the noise threshold information corresponding to the inter-frame difference image and the reference image;

[0203] 05: Determine the local feature weight based on the local feature information corresponding to the inter-frame difference image and the reference image;

[0204] 06: Determine the fusion weight based on the noise feature weight and the local feature weight;

[0205] 07: Fusion processing is performed on multiple frames of original images according to the fusion weight to obtain a fused image.

[0206] For another example, when the program 310 is executed by the processor 320, the following image processing method is implemented:

[0207] 021: Get the image brightness information of each frame of the original image;

[0208] 022: Calculate the image brightness mean of each frame of the original image based on the image brightness information;

[0209] 023: taking the frame with the largest average brightness value among the multiple frames of original images as the reference image; and / or

[0210] 024: Get the image gradient information of each frame of the original image;

[0211] 025: Calculate the image gradient mean of each frame of the original image based on the image gradient information;

[0212] 026: The frame with the largest image gradient mean among multiple frames of original images is used as the reference image.

[0213] It should be noted that the explanations of the image processing method and the image processing device 100 in the aforementioned embodiments are also applicable to the computer-readable storage medium 300 in the embodiments of the present application, and will not be elaborated here.

[0214] In summary, in the image processing method, image processing device 100, terminal 200, and computer-readable storage medium 300 of the embodiments of the present application, when selecting the fusion weight when fusing multiple frames of original images, on the one hand, the noise feature weight is determined based on the inter-frame difference image and the noise threshold information, and on the other hand, the local feature weight is determined based on the inter-frame difference image and the local feature information. Finally, the fusion weight is determined by combining the noise feature weight and the local feature weight. This can achieve a better multi-frame fusion effect and generate a high-quality fused image with clear images and no ghosting. In addition, the image processing method of the embodiments of the present application can also be applied to fields such as image super-resolution, noise reduction, and high dynamic range imaging (HDR).

[0215] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0216] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0217] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a computer-readable storage medium can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include the following: an electrical connection having one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner as necessary, and then stored in a computer memory.

[0218] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0219] Those skilled in the art will appreciate that all or part of the steps carried out in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment. In addition, the various functional units in the various embodiments of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk or an optical disk, etc.

[0220] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are illustrative and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. An image processing method, characterized in that: include: Get multiple frames of original images; Selecting one frame from the multiple frames of the original image as a reference image and using the remaining frames as input images; obtaining an inter-frame difference image according to the difference between the input image and the reference image; determining a noise feature weight according to noise threshold information corresponding to the inter-frame difference image and the reference image; Determining a local feature weight according to local feature information corresponding to the inter-frame difference image and the reference image; Determining a fusion weight according to the noise feature weight and the local feature weight; Performing a fusion process on the multiple frames of the original image according to the fusion weight to obtain a fused image; The noise threshold information is obtained by: obtaining an original noise threshold according to the light sensitivity when the original image is captured, obtaining image brightness information of the reference image, and determining the noise threshold information according to the image brightness information and the original noise threshold; The local feature information is obtained by calculating a local standard deviation of the reference image to obtain the local feature information.

2. The image processing method according to claim 1, wherein: The selecting one frame from the multiple frames of the original image as a reference image includes: Obtaining image brightness information of each frame of the original image; Calculating the image brightness mean of each frame of the original image according to the image brightness information; Using a frame with the largest average brightness among the multiple frames of the original image as the reference image; and / or Obtaining image gradient information of each frame of the original image; Calculating the image gradient mean of each frame of the original image according to the image gradient information; The frame with the largest image gradient mean among the multiple frames of the original image is used as the reference image.

3. The image processing method according to claim 1, wherein: The step of obtaining an inter-frame difference image according to a difference between the input image and the reference image includes: Calculating the difference between the input image and the reference image to obtain an initial difference image; The initial difference image is smoothed to obtain the inter-frame difference image.

4. The image processing method according to claim 1, wherein: The determining of the noise feature weight according to the noise threshold information corresponding to the inter-frame difference image and the reference image includes: performing noise threshold estimation on the reference image to obtain the noise threshold information; Performing a first motion detection according to a size relationship between the inter-frame difference image and the noise threshold information to determine an initial noise weight; The noise feature weight is obtained by performing morphological processing on the initial noise weight.

5. The image processing method according to claim 4, characterized in that The performing first motion detection according to the magnitude relationship between the inter-frame difference image and the noise threshold information to determine the initial noise weight includes: comparing the inter-frame difference image and the noise threshold information at corresponding pixel positions; When a difference value of the inter-frame difference image at the corresponding pixel position is greater than the noise threshold value of the noise threshold information at the corresponding pixel position, marking a region where the corresponding pixel position is located in the input image corresponding to the inter-frame difference image as a motion region; When a difference value of the inter-frame difference image at the corresponding pixel position is less than the noise threshold value of the noise threshold information at the corresponding pixel position, marking a region where the corresponding pixel position is located in the input image corresponding to the inter-frame difference image as a static region; The weight value of the motion area is set as a first initial noise weight, and the weight value of the static area is set as a second initial noise weight; wherein the first initial noise weight is smaller than the second initial noise weight.

6. The image processing method according to claim 1, wherein: The determining of the local feature weight according to the local feature information corresponding to the inter-frame difference image and the reference image includes: performing a second motion detection according to the difference value of the inter-frame difference image and the local standard deviation of the local feature information; The local feature weight is determined according to a result of the second motion detection.

7. The image processing method according to claim 1, wherein: The determining of the local feature weight according to the local feature information corresponding to the inter-frame difference image and the reference image includes: Calculating the local standard deviation of the reference image to obtain the local feature information; Constructing a feature weight relationship between the difference value of the inter-frame difference image, the local standard deviation of the local feature information, and the local feature weight; wherein, when the local standard deviation of the local feature information is constant, the difference value of the inter-frame difference image is negatively correlated with the local feature weight; and when the difference value of the inter-frame difference image is constant, the local standard deviation of the local feature information is positively correlated with the local feature weight; The local feature weight is determined according to the difference value of the inter-frame difference image, the local standard deviation of the local feature information, and the feature weight relationship.

8. The image processing method according to claim 3, wherein: The inter-frame difference image is obtained by smoothing the initial difference image through a first sampling window, the local feature information is obtained by calculating a local standard deviation of a second sampling window of the reference image, the range of the first sampling window is smaller than the range of the second sampling window, and the inter-frame difference image is obtained according to the difference between the input image and the reference image, further comprising: Calibration processing is performed on the inter-frame difference image so that the inter-frame difference image corresponding to the first sampling window matches the local feature information corresponding to the second sampling window.

9. The image processing method according to claim 1, wherein: The determining of the fusion weight according to the noise feature weight and the local feature weight includes: The product of the noise feature weight and the local feature weight is used as the fusion weight.

10. An image processing device, characterized in that: include: An acquisition module is used to acquire multiple frames of original images; A selection module, configured to select one frame from the multiple frames of the original image as a reference image and use the remaining frames as input images; a processing module, configured to obtain an inter-frame difference image according to a difference between the input image and the reference image; A first determining module, configured to determine a noise feature weight according to noise threshold information corresponding to the inter-frame difference image and the reference image; A second determining module is configured to determine a local feature weight based on local feature information corresponding to the inter-frame difference image and the reference image; A third determining module, configured to determine a fusion weight according to the noise feature weight and the local feature weight; A fusion module, configured to fuse multiple frames of the original image according to the fusion weight to obtain a fused image; The noise threshold information is obtained by: obtaining an original noise threshold according to the light sensitivity when the original image is captured, obtaining image brightness information of the reference image, and determining the noise threshold information according to the image brightness information and the original noise threshold; The local feature information is obtained by calculating a local standard deviation of the reference image to obtain the local feature information.

11. A terminal, characterized in that: The terminal includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the image processing method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image processing method according to any one of claims 1 to 9 is implemented.

Images