Image processing method, image processing device, terminal and readable storage medium

Abstract

The present application discloses an image processing method, an image processing device, a terminal, and a non-volatile computer-readable storage medium. The image processing method includes: acquiring N frames of images, wherein N ≥ 2; determining a reference image in the N frames of images, and the remaining N-1 frames of images are images to be processed; generating N frames of de-ghosted images according to the reference image and the images to be processed; and fusing the N frames of the de-ghosted images to obtain a frame of fused image. In the image processing method, image processing device, terminal, and non-volatile computer-readable storage medium of the present application, since a selected reference image is used when generating a de-ghosted image, and the fusion process is performed on N frames of de-ghosted images, the role of the reference image is fully utilized, so that the ghost of the fused image is eliminated more thoroughly, thereby improving the clarity of the fused image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to an image processing method, an image processing device, a terminal, and a non-volatile computer-readable storage medium. Background Art

[0002] With the continuous development of image acquisition devices such as mobile phones, charge coupled device (CCD) cameras, and complementary metal oxide semiconductor (CMOS) cameras, people's demand for high-resolution images is growing, and the pursuit of high-quality visual experience is getting higher and higher. In order to improve the performance of CCD or CMOS camera sensors, many super-resolution algorithms have been proposed in the industry. Multi-image super-resolution reconstruction is a technology widely used in the industry. It uses the complementary information between multiple frames of images and fuses the information in a higher resolution grid to reconstruct details and improve image clarity.

[0003] However, due to camera motion or the presence of moving objects relative to the background in the image, there will be some overlap in the fusion results of multiple images, which greatly reduces the image quality and affects subsequent evaluation and observation. This phenomenon of pixel overlap and misalignment in the fusion results is called "ghosting" artifacts. How to remove "ghosting" and avoid image blur has become a difficult problem that technicians in this field need to solve urgently. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, an image processing device, a terminal and a non-volatile computer-readable storage medium, which are used to at least solve the problem of how to remove "ghost images" and avoid image blur.

[0005] The image processing method of the embodiment of the present application includes: acquiring N frames of images, where N≥2; determining a reference image frame among the N frames of images, and the remaining N-1 frames of images are images to be processed; generating N frames of de-ghosted images based on the reference image and the images to be processed; and fusing the N frames of de-ghosted images to obtain a frame of fused image.

[0006] The image processing device of the embodiment of the present application includes an acquisition module, a determination module, a generation module and a fusion module. The acquisition module is used to acquire N frames of images, where N≥2; the determination module is used to determine a reference image in the N frames of images, and the remaining N-1 frames of images are images to be processed; the generation module is used to generate N frames of ghost-free images based on the reference image and the image to be processed; and the fusion module is used to fuse the N frames of ghost-free images to obtain a frame of fused image.

[0007] The terminal of an embodiment of the present application includes one or more processors, a memory and one or more programs, wherein one or more of the programs are stored in the memory and executed by one or more of the processors, and the program includes a method for executing the following image processing: acquiring N frames of images, wherein N≥2; determining a frame of reference image among the N frames of images, and the remaining N-1 frames of images are images to be processed; generating N frames of de-ghosted images based on the reference image and the images to be processed; and fusing the N frames of de-ghosted images to obtain a frame of fused image.

[0008] The non-volatile computer-readable storage medium storing a computer program in an embodiment of the present application implements the following image processing method when the computer program is executed by one or more processors: acquiring N frames of images, where N≥2; determining a reference image frame among the N frames of images, and the remaining N-1 frames of images are images to be processed; generating N frames of de-ghosted images based on the reference image and the images to be processed; and fusing the N frames of de-ghosted images to obtain a frame of fused image.

[0009] In the image processing method, image processing device, terminal and non-volatile computer-readable storage medium of the implementation mode of the present application, after a reference image is determined in N frames of images and the remaining N-1 frames of images are images to be processed, N frames of de-ghosted images are first generated according to the reference image and the images to be processed, and finally the N frames of de-ghosted images are fused to obtain a frame of fused image. Since the selected reference image is used when generating the de-ghosted image, and the fusion process is performed on the N frames of de-ghosted images, the role of the reference image is fully utilized, so that the ghost of the fused image is eliminated more thoroughly, thereby improving the clarity of the fused image.

[0010] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0012] Figure 1 is a flowchart of an image processing method of certain embodiments of the present application;

[0013] Figure 2 is a schematic diagram of the structure of an image processing device in some embodiments of the present application;

[0014] Figure 3 is a schematic diagram of the structure of a terminal in some implementation modes of the present application;

[0015] Figure 4 is a schematic diagram of the internal architecture of a terminal for taking photos according to certain embodiments of the present application;

[0016] Figure 5 is a flowchart of an image processing method of certain embodiments of the present application;

[0017] Figure 6 It is a schematic diagram of the principle of determining a reference image and an image to be processed in an image processing method in certain embodiments of the present application;

[0018] Figures 7 to 9 is a flowchart of an image processing method of certain embodiments of the present application;

[0019] Fig.10 and Fig.11 It is a schematic diagram of the principle of performing noise sampling on a reference image and / or a registered image in the image processing method in certain embodiments of the present application;

[0020] Fig.12 It is a schematic diagram of the principle of obtaining the N-1 frame noise difference map in the image processing method of certain embodiments of the present application;

[0021] Fig.13 and Fig.14 is a flowchart of an image processing method of certain embodiments of the present application;

[0022] Fig.15 It is a schematic diagram of the principle of obtaining an N-1 frame pixel difference map in an image processing method in certain embodiments of the present application;

[0023] Fig.16 It is a schematic diagram of the principle of obtaining the noise characteristic value of each pixel in the image processing method of certain embodiments of the present application;

[0024] Fig.17 It is a schematic diagram of the principle of obtaining a noise feature map in an image processing method in certain embodiments of the present application;

[0025] Fig.18 It is a schematic diagram of the principle of obtaining a first weight map in an image processing method in certain embodiments of the present application;

[0026] Fig.19 It is a schematic diagram of the principle of obtaining a second weight map in the image processing method in certain embodiments of the present application;

[0027] Fig. 20 It is a schematic diagram of the principle of obtaining a second weight map from a noise feature map in an image processing method in certain embodiments of the present application;

[0028] Fig.21is a flowchart of an image processing method of certain embodiments of the present application;

[0029] Fig. 22 It is a schematic diagram of the principle of obtaining N frames of ghost-free images in the image processing method of certain embodiments of the present application;

[0030] Fig.23 is a flowchart of an image processing method of certain embodiments of the present application;

[0031] Fig.24 It is a schematic diagram of the principle of obtaining a sum value graph in the image processing method in certain embodiments of the present application;

[0032] Fig.25 It is a schematic diagram of the principle of obtaining a fused image in the image processing method in certain embodiments of the present application;

[0033] Fig.26 It is a comparison diagram of a fused image obtained by the image processing method of the present application and a fused image obtained by a common multi-frame fusion algorithm;

[0034] Fig. 27 It is a schematic diagram of the connection status of a non-volatile computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0035] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0036] At present, due to camera motion or the presence of moving objects relative to the background in the image, there will be a certain overlap in the fusion results of multiple images, which greatly reduces the image quality and affects subsequent evaluation and observation. This phenomenon of pixel overlap and misalignment in the fusion result is called "ghosting" artifacts. How to remove "ghosting" and avoid image blur has become a difficult problem that technicians in this field need to solve urgently. To solve this problem, the present application provides an image processing method and an image processing device 10( Figure 2 As shown), a terminal 100 ( Figure 3 ) and a non-volatile computer-readable storage medium ( Fig. 27 shown).

[0037] See also Figure 1 The image processing method of the embodiment of the present application includes:

[0038] 01: Get N frames of images, where N ≥ 2;

[0039] 03: Determine a reference image among N frames of images, and the remaining N-1 frames of images are images to be processed;

[0040] 05: Generate N frames of ghost-free images based on the reference image and the image to be processed; and

[0041] 07: Perform fusion processing on N frames of de-ghosted images to obtain a fused image.

[0042] See also Figure 2 The above-mentioned image processing method can be applied to an image processing device 10. The image processing device 10 of the embodiment of the present application includes an acquisition module 11, a determination module 13, a generation module 15 and a fusion module 17. The acquisition module 11 is used to acquire N frames of images, where N≥2. The determination module 13 is used to determine a reference image in the N frames of images, and the remaining N-1 frames of images are images to be processed. The generation module 15 is used to generate N frames of de-ghosted images based on the reference image and the image to be processed; the fusion module 17 is used to fuse the N frames of de-ghosted images to obtain a frame of fused image.

[0043] See also Figure 3 , the above-mentioned image processing method can be applied to the terminal 100. The terminal 100 of one embodiment of the present application includes a body 20, one or more processors 40, a memory 30, and one or more programs. Among them, one or more processors 40 and the memory 30 are installed in the body 20, one or more programs are stored in the memory 30, and are executed by one or more processors 40, and the program includes a program for executing the image processing methods in 01, 03, 05 and 07. That is, one or more processors 40 are used to obtain N frames of images, where N≥2; determine a frame of reference image in the N frames of images, and the remaining N-1 frames of images are images to be processed; generate N frames of de-ghosted images according to the reference image and the image to be processed; and fuse the N frames of de-ghosted images to obtain a frame of fused image.

[0044] A terminal 100 according to another embodiment of the present application may include a main body 20 and an image processing device 10 according to an embodiment of the present application, and the image processing device 10 is installed in the main body 20 .

[0045] Specifically, see Figure 4, which is a schematic diagram of the internal architecture of the terminal 100 for taking photos, showing the entire process of taking photos. When the user clicks on the photo APP and selects the photo mode and photo parameters in the photo APP, a "burstCapture" instruction is issued to the image sensor of the hardware abstraction layer (HAL) layer. The sensor of the HAL layer responds to the "quick capture" instruction, and performs a photo-taking action according to at least part of the photo-taking parameters to obtain multiple frames of YUV images, and transmits them to the algorithm post-processing module (Algo Process Service, APS) through the photo-taking APP. The APS performs a multi-frame fusion algorithm on the multiple frames of YUV images to obtain a frame of fused YUV image, and executes a YUV to JPEG algorithm on the fused YUV image. When executing the YUV to JPEG algorithm, the APS transmits the fused YUV image to the image signal processor (Image Signal Processing, ISP) of the HAL layer. The image processing engine (Image Process Engine, IPE) in the APS performs format conversion on the fused YUV image to obtain a JPEG image. The encoder (Encoder) in the APS encodes and compresses the JPEG and returns the encoded and compressed JPEG image to the APS. The storage unit in the APS stores the encoded and compressed JPEG image. The subsequent gallery APP can retrieve the encoded and compressed JPEG image from the storage unit and decode and decompress it, thereby displaying the decompressed JPEG image and presenting it to the user for viewing and appreciation. Thus, in some embodiments, the image processing device 10 may be a module integrated in an APS. One or more processors 40 in the terminal 100 may be an APS.

[0046] Among them, the shooting mode includes but is not limited to video, photo, portrait, night scene, text and other modes. The shooting parameters include metadata corresponding to the multi-frame YUV image. The metadata includes relevant information of the data to be processed, such as 3a (automatic exposure control AE, automatic focus control AF, automatic white balance control AWB) information, picture information (width and height of the picture, number) parameters, exposure parameters (aperture size, shutter speed and sensitivity aperture value), black level correction parameters, and shadow correction (Lens Shading Correction, LSC) parameters.

[0047] In one embodiment, the data to be processed is a multi-frame YUV image initially acquired by the sensor. At this time, after receiving the metadata, the APS can post-process each frame of the multi-frame YUV image according to the metadata, and the post-processing can be performed according to a parameter in the metadata or according to any multiple parameters in sequence. For example, the APS adjusts the image brightness corresponding to each frame of the YUV image according to the ISO value in the metadata. It should be noted that in the embodiment of the present application, the exposure parameters of the multi-frame YUV images initially acquired by the sensor remain consistent, thereby ensuring easy registration and alignment during subsequent processing. Of course, other parameters of the multi-frame YUV images initially acquired by the sensor can also remain consistent, such as 3a parameters, black level correction parameters and LSC parameters, which are not listed here one by one. At the same time, the sensor initially acquires multi-frame YUV images and can also add anti-shake and focus functions to ensure that the multi-frame YUV images are clear and avoid the original image from being out of focus, blurring, etc.

[0048] In another embodiment, the data to be processed is a frame of YUV image processed by a multi-frame fusion algorithm. In this case, after receiving the metadata, the APS can perform post-processing on the fused YUV image according to the metadata, and the post-processing can be performed according to one parameter in the metadata or any multiple parameters in sequence. For example, the APS performs black level correction on the fused YUV image for each frame of the YUV image according to the black level correction parameter in the metadata, and performs shadow correction on the fused YUV image for each frame of the YUV image according to the LSC parameter in the metadata.

[0049] In the image processing method, image processing device 10 and terminal 100 of the present application, after a reference image is determined in N frames of images and the remaining N-1 frames of images are images to be processed, N frames of de-ghosted images are first generated according to the reference image and the images to be processed, and finally the N frames of de-ghosted images are fused to obtain a frame of fused image. Since the selected reference image is used when generating the de-ghosted image, and the fusion process is performed on the N frames of de-ghosted images, the role of the reference image is fully utilized, so that the ghost of the fused image is eliminated more thoroughly, thereby improving the clarity of the fused image.

[0050] See also Figure 5 In some implementations, 03: determining a reference image in N frames of images, comprising:

[0051] 031: Determine a reference image from N frames according to the clarity.

[0052] The determination module 13 is further configured to determine a reference image in the N frames of images according to the definition.

[0053] One or more programs are executed by one or more processors 40, and the programs also include a method for executing the image processing method in 031. That is, the processor 40 is also used to determine a reference image in N frames of images according to the definition. Specifically, the processor 40 is also used to select a frame of image with the largest definition value from the N frames of images as a reference image.

[0054] Specifically, the frame with the largest definition value among N frames of images can be used as the reference image. Figure 6 , assuming that N=4, that is, the acquisition module 11 or the processor 40 acquires 4 frames of images, and calculates the clarity values ​​of the 4 frames of images respectively, takes the frame with the largest clarity value as the reference image, and the remaining 3 frames as the images to be processed. Among them, the clarity value of the image can be represented by the proportion of the number of pixels of high-frequency information in the image to all pixels in the image. The higher the proportion, the greater the image clarity value. For example, if the number of pixels of high-frequency information in an image accounts for 20% of the number of all pixels in the image, the clarity value of the image is represented by the proportion of 20%.

[0055] See also Figure 7 In some embodiments, 05: generating N frames of de-ghosted images according to the reference image and the image to be processed, including:

[0056] 051: Register each frame of the image to be processed with the reference image to determine N-1 frames of registered images;

[0057] 053: Perform noise estimation on each frame of the registered image and determine N-1 noise levels;

[0058] 055: Perform motion detection on each frame of the registered image to determine N-1 first weight maps;

[0059] 057: Perform morphological operations on each first weight map to determine N-1 second weight maps; and

[0060] 059: Obtain N frames of de-ghosted images according to N-1 second weight maps, a preset reference weight map, N-1 frames of registered images, and a reference image.

[0061] Please combine Figure 2 The generation module 15 is also used to: align each frame of the image to be processed with the reference image to determine N-1 frames of aligned images; perform noise estimation on each frame of the aligned images to determine N-1 noise levels; perform motion detection on each frame of the aligned images to determine N-1 first weight maps; perform morphological operations on each first weight map to determine N-1 second weight maps; and obtain N frames of de-ghosted images based on the N-1 second weight maps, the preset reference weight map, the N-1 frames of aligned images, and the reference image.

[0062] Please combine Figure 3, one or more programs are executed by one or more processors 40, and the programs also include methods for executing the image processing methods in 051, 053, 055, 057 and 059. That is, the processor 40 is also used to align each frame of the image to be processed with the reference image to determine N-1 frames of aligned images; perform noise estimation on each frame of aligned images to determine N-1 noise levels; perform motion detection on each frame of aligned images to determine N-1 first weight maps; perform morphological operations on each first weight map to determine N-1 second weight maps; and obtain N frames of ghost-free images according to the N-1 second weight maps, the preset reference weight map, the N-1 frames of aligned images and the reference image.

[0063] In the image processing method, image processing device 10 and terminal 100 in the present application, the first weight map is generated based on noise estimation, that is, the noise level of the registered image is first estimated, and then the noise level is used as a factor to control the generation of the first weight map. Compared with the traditional method of obtaining the weight map through threshold judgment, it can avoid misjudgment of dark and light flat areas. In addition, considering that rough threshold segmentation can easily cause an obvious sense of fragmentation in the image, the present application also uses morphological operations on the first weight map to obtain the second weight map, so that the second weight map forms a connected domain, and finally the ghost removal of the ghost removal image obtained according to the second weight map is more thorough, and the fused image generated after the final fusion processing is also smoother, and the fusion effect is better.

[0064] See also Figure 8 Specifically, in some embodiments, 051: registering each frame of the image to be processed with the reference image to determine N-1 frames of registered images may include:

[0065] 0511: Extract feature points from each frame of the image to be processed and the reference image to obtain multiple feature points, and describe the features of each feature point;

[0066] 0513: Match the feature points of each frame of the image to be processed with the feature points of the reference image to obtain multiple feature point pairs;

[0067] 0515: solving the transformation matrix between each frame of the image to be processed and the reference image in the matched feature point pairs; and

[0068] 0517: Align the corresponding image to be processed with the reference image according to the transformation matrix to obtain N-1 frames of registered images.

[0069] Please combine Figure 2The generation module 15 is also used for: extracting feature points of each frame of the image to be processed and the reference image to obtain multiple feature points, and describing the features of each feature point; matching the feature points of each frame of the image to be processed with the feature points of the reference image to obtain multiple feature point pairs; and solving the transformation matrix between each frame of the image to be processed and the reference image in the matched feature point pairs; and aligning the corresponding image to be processed with the reference image according to the transformation matrix to obtain N-1 frames of registered images.

[0070] Please combine Figure 3 , one or more programs are executed by one or more processors 40, and the programs also include the image processing methods for executing 0511, 0513, 0515 and 0517. That is, the processor 40 is also used to extract feature points from each frame of the image to be processed and the reference image to obtain multiple feature points, and to describe the features of each feature point; match the feature points of each frame of the image to be processed with the feature points of the reference image to obtain multiple feature point pairs; solve the transformation matrix between each frame of the image to be processed and the reference image in the matched feature point pairs; and align the corresponding image to be processed with the reference image according to the transformation matrix to obtain N-1 frame registered images.

[0071] Among them, feature points are extracted from each frame of the image to be processed and the reference image to obtain multiple feature points. The multiple feature points can be obtained by using the same method or by using different methods. If the multiple feature points are obtained by using the same method, it becomes easier to determine multiple feature point pairs in 0513.

[0072] Furthermore, in some implementations, 051: registering each frame of the image to be processed with the reference image to determine N-1 frames of registered images, may also include: 0514: using the RANSAC algorithm to remove bad pixels.

[0073] The generation module 15 is also used to: remove bad pixels using the RANSAC algorithm. One or more programs are executed by one or more processors 40, and the program also includes a method for executing the image processing method in 0514. That is, the processor 40 is also used to remove bad pixels using the RANSAC algorithm. At this time, the RANSAC algorithm can be used to remove bad pixels first. When executing 0515, the transformation matrix between each frame of the image to be processed and the reference image is solved by using the feature point pairs obtained by matching after removing the bad pixels.

[0074] In the image processing method, image processing device 10 and terminal 100 of the present application, each frame of the image to be processed is first aligned with the reference image to obtain N-1 frames of aligned images, and then the N-1 frames of aligned images and the reference images are used to perform noise estimation, motion detection, morphological operations, etc., so that the misalignment and superposition of the image caused by camera movement or object movement during noise estimation, motion detection, and morphological operations are minimized, thereby ensuring the accuracy of noise estimation, motion detection, morphological operations, etc. and the best results.

[0075] More specifically, see Fig. 9 In some embodiments, 053: performing noise estimation on each frame of the registered image to determine N-1 noise levels may include:

[0076] 0531: Perform noise sampling on the reference image and the N-1 frame registration image in units of M*M pixels to obtain a reference sampling image corresponding to the reference image and an N-1 frame registration sampling image corresponding to the N-1 frame registration image, respectively, 2≤M≤15% of the total number of pixels in the length direction;

[0077] 0533: Obtaining a noise difference map of frame N-1 according to the noise difference value between the corresponding pixel positions of the reference sampling map and each frame of the registration sampling map; and

[0078] 0535: Obtain N-1 noise levels according to the N-1 frame noise difference map and the frame number N.

[0079] Please combine Figure 2 The generation module 15 is also used for: performing noise sampling on the reference image and the N-1 frame registration images in units of M*M pixels to obtain a reference sampling map corresponding to the reference image and an N-1 frame registration sampling map corresponding to the N-1 frame registration image, respectively, 2≤M≤15% of the total number of pixels in the length direction; obtaining an N-1 frame noise difference map according to the noise difference value between the corresponding pixel positions of the reference sampling map and each frame registration sampling map; and obtaining N-1 noise levels according to the N-1 frame noise difference map and the frame number N.

[0080] Please combine Figure 3 , one or more programs are executed by one or more processors 40, and the programs also include methods for executing the image processing methods in 0531, 0533 and 0535. That is, the processor 40 is also used to perform noise sampling on the reference image and the N-1 frame registration images in units of M*M pixels to obtain a reference sampling map corresponding to the reference image and an N-1 frame registration sampling map corresponding to the N-1 frame registration image, respectively, 2≤M≤15% of the total number of pixels in the length direction; obtain an N-1 frame noise difference map according to the noise difference value between the reference sampling map and the corresponding pixel position of each frame registration sampling map; and obtain N-1 noise levels according to the N-1 frame noise difference map and the frame number N.

[0081] Specifically, when performing noise sampling on the reference image or the registered image, an estimated sampling method is used. Therefore, it is not necessary to take the noise values ​​of all pixels. Instead, the noise values ​​of some pixels can be taken according to certain rules to obtain a reference sampling map or a registered sampling map. For example, equal-interval sampling is used. Therefore, for the reference image, the value of M can be greater than or equal to 2, and less than or equal to 15% of the total number of pixels in the length direction of the reference image. Preferably, 5% of the total number of pixels in the length direction of the reference image ≤ M ≤ 15% of the total number of pixels in the length direction of the reference image. For the registered image, the value of M can be greater than or equal to 2, and less than or equal to 15% of the total number of pixels in the length direction of the registered image. Preferably, 5% of the total number of pixels in the length direction of the registered image ≤ M ≤ 15% of the total number of pixels in the length direction of the registered image. When the value of M is less than 5% of the total number of pixels in the image (reference image / registered image) length, the interval unit of interval sampling is too small, and the speed is not much different from the speed of sampling all pixels, which cannot achieve the effect of noise estimation; when the value of M is greater than 15% of the total number of pixels in the image (reference image / registered image) length, the interval unit of interval sampling is too large and cannot well represent the image characteristics.

[0082] See also Fig.10 and Fig.11 For an image with a total pixel of 100*100, assuming M=4, it means that 4 is used as the reference, which is equivalent to 4*4=16 pixels as a sampling unit for sampling (a dotted box is a sampling unit), there are 25 sampling units in the length direction, and 25 sampling units in the width direction. The final sampling map generated is a 25*25 array, which can be used for noise estimation well, and the noise level obtained can also represent the image. However, if it is also for an image with a total pixel of 100*100, if M=50, it means that 50 is used as the reference, which is equivalent to 50*50=250 pixels as a sampling unit for sampling, there are only 2 sampling units in the length direction, and there are only 2 sampling units in the width direction. The final sampling map generated is a 2*2 array, the graphic block is relatively large, and the noise level obtained is not representative. Therefore, setting 5% of the total number of pixels in the reference image length direction ≤ M ≤ 15% of the total number of pixels in the reference image length direction and 5% of the total number of pixels in the registration image length direction ≤ M ≤ 15% of the total number of pixels in the registration image length direction not only plays the original role of noise estimation, but also the noise level can well represent the image characteristics.

[0083] In some embodiments, see Fig.10 , the noise value of each pixel of the reference sampling image may be the noise value of the pixel at the same position in the corresponding sampling unit. Specifically, Fig.10The noise value of pixel 11 in the reference sampling image shown in the right figure can be the noise value of any pixel (any pixel from pixel 11 to pixel 44) in the first sampling unit on the first row of the reference image on the left. For example, the noise value of pixel 11 in the reference sampling image is the noise value of pixel 11 in the reference image. Fig.10 The noise value of pixel 12 in the reference sampling image shown in the right figure can be the noise value of the pixels corresponding to the position of pixel 11 in the reference image in pixels 15-pixel 48 in the second sampling unit on the first row of the reference image on the left, that is, the noise value of pixel 12 in the reference sampling image is the noise value of pixel 15 in the reference image. Fig.10 The noise value of pixel 21 in the reference sampling image shown in the right figure is the noise value of pixel 51 in the reference image. Fig.10 The noise value of pixel 22 in the reference sampling image shown in the right figure is the noise value of pixel 55 in the reference image. By analogy, the noise value of each pixel in the entire reference sampling image can be obtained. Fig.10 As shown, the noise value of each pixel of the registration sampling image may also be the noise value of the pixel at the same position in the corresponding sampling unit. The specific sampling method is the same as the sampling method of the reference sampling image, which will not be described in detail here.

[0084] In some further embodiments, see Fig.11 , the noise value of each pixel of the reference sampling image may be the average of the noise values ​​of all pixels in the corresponding sampling unit. Specifically, Fig.11 The noise value of pixel 11 in the reference sampling image shown in the right figure may be the average of the noise values ​​of all pixels (pixel 11 to pixel 44) in the first sampling unit on the first row of the reference image on the left. Fig.11 The noise value of pixel 12 in the reference sampling image shown in the right figure may be the average of the noise values ​​of all pixels (pixel 15 to pixel 48) in the second sampling unit on the first row of the reference image on the left. Fig.11 The noise value of pixel 21 in the reference sampling image shown in the right figure may be the average of the noise values ​​of all pixels (pixel 51 to pixel 84) in the first sampling unit on the second row of the reference image on the left. Fig.11 The noise value of pixel 22 in the reference sampling image shown in the right figure can be the average of the noise values ​​of all pixels (pixel 55 to pixel 88) in the second sampling unit on the second row of the reference image on the left. By analogy, the noise value of each pixel in the entire reference sampling image can be obtained. Fig.11 As shown, the noise value of each pixel of the registration sampling image can also be the average of the noise values ​​of all pixels in the corresponding sampling unit. The specific sampling method is the same as the sampling method of the reference sampling image, which will not be repeated here.

[0085] In noise estimation, noise sampling is performed using M*M pixels as a unit. If there are extreme points in a larger area, the final effect may be affected. That is, if there are obvious light and dark area blocks in the image, the calculated noise difference value will be between the two and is not representative. Therefore, in some other embodiments, the noise value of each pixel of the reference sampling image can also be the variance value of the noise values ​​of all pixels in the corresponding sampling unit. Similarly, the noise value of each pixel of the registration sampling image can also be the variance value of the noise values ​​of all pixels in the corresponding sampling unit, thereby ensuring that the registration sampling image and the reference sampling image are representative.

[0086] See also Fig.12 , is a schematic diagram of the principle of obtaining three noise difference maps based on the noise difference values ​​between the corresponding pixel positions of one reference sampling map P and three registration sampling maps (A, B, C) (assuming N=4). In the noise difference map, the noise difference value of pixel ij is the noise value of pixel ij in the registration sampling map minus the noise value of pixel ij in the reference sampling map, that is, the noise difference value ij of the noise difference map = the noise value ij of the registration sampling map - the noise value ij of the reference sampling map, where i is the row number and j is the column number. For example, the noise difference value formed by the registration sampling map A and the reference sampling map P Figure 1 For example, the noise difference Figure 1 The noise difference value of pixel 11 in the registration sampling image A minus the noise value of pixel 11 in the reference sampling image P is A11-P11. Figure 1 The noise difference value of pixel 12 in the registration sampling image A minus the noise value of pixel 12 in the reference sampling image P is A12-P12. Figure 1 The noise difference value of pixel 13 in the registration sampling image A minus the noise value of pixel 13 in the reference sampling image P is A13-P13. Figure 1 The noise difference value of pixel 14 in the registration sampling image A minus the noise value of pixel 14 in the reference sampling image P is A14-P14. Figure 1 The noise difference between the registration sample image B and the reference sample image P is calculated in the same way as the other pixels in Figure 2 , the noise difference between the registration sampling image C and the reference sampling image P Figure 3 They are all generated according to this method and will not be described in detail here.

[0087] See also Fig.13 In some implementations, 0535: obtaining N-1 noise levels according to the N-1 frame noise difference graph and the frame number N may include:

[0088] 05351: Calculate the overall difference value of the noise difference map of each frame to obtain N-1 overall difference values;

[0089] 05353: Get the median of N-1 overall difference values; and

[0090] 05355: The ratio of the median to the number of frames N is taken as the noise level α.

[0091] Please combine Figure 2 The generating module 15 is further used for: calculating the overall difference value of the noise difference map of each frame to obtain N-1 overall difference values; obtaining the median of the N-1 overall difference values; and taking the ratio of the median to the number of frames N as the noise level α.

[0092] Please combine Figure 3 , one or more programs are executed by one or more processors 40, and the programs also include a method for executing the image processing methods in 05351, 05353, and 05355. That is, the processor 40 is also used to calculate the overall difference value of the noise difference map of each frame to obtain N-1 overall difference values; obtain the median of the N-1 overall difference values; and use the ratio of the median to the number of frames N as the noise level α.

[0093] For details, please refer to Fig.12 , the noise difference between the registration sampling image B and the reference sampling image P Figure 1 The overall difference value AP = (A11-P11)+(A12-P12)+(A13-P13)+(A14-P14)+(A21-P21)+(A22-P22)+(A23-P23)+(A24-P24)+(A31-P31)+(A32-P32)+(A33-P33)+(A34-P34)+(A41-P41)+(A42-P42)+(A43-P43)+(A44-P44), the noise difference between the registration sampling image B and the reference sampling image P Figure 2 The overall difference value BP = (B11-P11) + (B12-P12) + (B13-P13) + (B14-P14) + (B21-P21) + (B22-P22) + (B23-P23) + (B24-P24) + (B31-P31) + (B32-P32) + (B33-P33) + (B34-P34) + (B41-P41) + (B42-P42) + (B43-P43) + (B44-P44), the noise difference between the registration sampling image C and the reference sampling image P Figure 3The overall difference value CP = (C11-P11) + (C12-P12) + (C13-P13) + (C14-P14) + (C21-P21) + (C22-P22) + (C23-P23) + (C24-P24) + (C31-P31) + (C32-P32) + (C33-P33) + (C34-P34) + (C41-P41) + (C42-P42) + (C43-P43) + (C44-P44). Calculate the median between the overall difference values ​​AP, BP and CP. Then take the ratio of the median to the number of frames N as the noise level α. Assuming the median is CP, the noise level α of these 4 frames is CP / 4.

[0094] See also Fig.14 In some embodiments, 055: performing motion detection on each frame of the registered image to obtain N-1 first weight maps, including:

[0095] 0551: Obtain the N-1 frame pixel difference map according to the pixel difference between the corresponding pixel positions of the reference image and each frame of the registered image;

[0096] 0553: amplifying the pixel difference in each frame of the pixel difference map according to the noise level to obtain the N-1 frame noise feature map; and

[0097] 0555: Perform threshold detection on the feature values ​​of all pixels in each frame of the noise feature map, and set an initial weight value for each pixel to obtain N-1 first weight maps.

[0098] Please combine Figure 2 The generation module 15 is also used to: obtain N-1 frames of pixel difference maps according to the pixel difference between the corresponding pixel positions of the reference image and each frame of the registered image; amplify the pixel difference in each frame of the pixel difference map according to the noise level to obtain N-1 frames of noise feature maps; and perform threshold detection on the characteristic values ​​of all pixels in each frame of the noise feature map, and set an initial weight value for each pixel to obtain N-1 first weight maps.

[0099] Please combine Figure 3 , one or more programs are executed by one or more processors 40, and the programs also include the method for executing the image processing methods in 0551, 0553 and 0555. That is, the processor 40 is also used to obtain N-1 frame pixel difference maps according to the pixel difference between the corresponding pixel positions of the reference image and each frame of the registered image; amplify the pixel difference in each frame of the pixel difference map according to the noise level to obtain N-1 frame noise feature maps; and perform threshold detection on the feature values ​​of all pixels in each frame of the noise feature map, and set an initial weight value for each pixel to obtain N-1 first weight maps.

[0100] See also Fig.15 and Fig.16 , is a schematic diagram of the principle of obtaining three-frame pixel difference maps based on the pixel difference between the corresponding pixel positions of one frame of reference image P' and three frames of registered images (A', B', C') (assuming N = 4). In the pixel difference map, the pixel difference of pixel ij is the absolute value of the pixel value of pixel ij in the registered image minus the pixel value of pixel ij in the reference image, that is, the pixel difference ij of the pixel difference map = |registered image pixel value ij-reference image pixel value ij|, where i is the row number and j is the column number. For example, the pixel difference formed by the registered image A' and the reference image P' Figure 1 For example, the pixel difference Figure 1 The pixel difference of pixel 11 in the registration image A' is the absolute value of the pixel value of pixel 11 in the reference image P' minus the pixel value of pixel 11 in the reference image P', that is, |A'11-P'11|. Figure 1 The pixel difference of pixel 12 in the registration image A' is the absolute value of the pixel value of pixel 12 in the reference image P' minus the pixel value of pixel 12 in the reference image P', that is, |A'12-P'12|. Figure 1 The pixel difference of pixel 13 in the registration image A' is the absolute value of the pixel value of pixel 13 in the reference image P' minus the pixel value of pixel 13 in the reference image P', that is, |A'13-P'13|. Figure 1 The pixel difference of pixel 14 in the registration image A' is the absolute value of the pixel value of pixel 14 minus the pixel value of pixel 14 in the reference image P', that is, |A'14-P'14|. Figure 1 The same calculation is done for other pixels in the image. The pixel difference between the registered image B' and the reference image P' Figure 2 , the pixel difference between the registered image C' and the reference image P' Figure 3 They are all generated according to this method and will not be described in detail here.

[0101] See also Fig.17 , is the pixel difference of one frame according to the noise level α Figure 1 The pixel difference (absolute value) in is amplified to obtain the corresponding noise characteristics Figure 1 Schematic diagram of the principle. The pixel difference between the other two frames Figure 2 and pixel differences Figure 3 The pixel difference (absolute value) in is amplified to obtain the corresponding noise feature map (assuming N = 4). Figure 1 In the example, the feature value of pixel ij is the pixel difference Figure 1 The absolute value of the pixel difference of pixel ij in is multiplied by the noise level α, that is, the noise characteristic Figure 1 The eigenvalue of pixel ij in the image = |pixel difference Figure 1The characteristic value of pixel ij in |*α, where i is the row number and j is the column number. For example, the noise characteristic Figure 1 The characteristic value of pixel 11 in the image is equal to |A'11-P'11|*α. The present application uses the noise level to amplify the pixel difference map to obtain a noise characteristic map, thereby increasing the robustness of the algorithm to noise.

[0102] In some embodiments, 0555: performing threshold detection on the characteristic values ​​of all pixels in each frame of the noise feature map, and setting an initial weight value for each pixel, may include: in the noise feature map, when the characteristic value of a pixel is greater than a preset threshold, determining that the position of the pixel is a motion area, and setting the initial weight value to 0; and in the noise feature map, when the characteristic value of a pixel is less than or equal to a preset threshold, determining that the position of the pixel is a non-motion area, and setting the initial weight value to 1.

[0103] Among them, the preset threshold is an adjustable parameter, which can be a value obtained based on experience. Different thresholds will be configured for different scenes. For example, the noise level in dark scenes is high, so the corresponding preset threshold will be higher. When the pixel difference is small, it will be considered to be caused by noise, not motion. Only when the difference is large to a certain extent, it is considered to be motion. In bright scenes, the noise is very small, and the corresponding preset threshold will be relatively low. Therefore, any slightly larger difference can be regarded as motion. In an example, the brighter the scene, the larger the value of the corresponding preset threshold, and the smaller the brightness of the scene, the smaller the value of the corresponding preset threshold.

[0104] Specifically, see Fig.18 , noise characteristics Figure 1 In the figure: if the feature values ​​of pixels 11, 21, 31, 41, 23, and 33 are all greater than the preset threshold, then the positions of pixels 11, 21, 31, 41, 23, and 33 are determined to be motion areas, and the initial weight value is set to 0; if the feature values ​​of pixels 12, 13, 14, 22, 24, 32, 34, 42, 43, and 44 are all less than or equal to the preset threshold, then the positions of pixels 12, 13, 14, 22, 24, 32, 34, 42, 43, and 44 are determined to be non-motion areas, and the initial weight value is set to 1, and the noise feature is obtained. Figure 1 The corresponding first weight Figure 1 . Get the noise characteristics Figure 2 The corresponding first weight Figure 2 It is also a similar method, and obtains noise characteristics Figure 3 The corresponding first weight Figure 3It is also a similar method, which will not be repeated here. It can be understood that the current weight value uses 0-1 binary segmentation, which is a bit rough. More weight values ​​can be set, such as dividing the feature value within the threshold range into 5 levels. When the feature value of the pixel in the noise feature map is less than 1 / 5 of the threshold, the weight value is 1, and when the feature value of the pixel is less than 2 / 5 of the threshold, the weight value is 4 / 5, etc. In addition, the weight value can also be set adaptively.

[0105] In some embodiments, the morphological operation includes an erosion process and / or a dilation process. Fig.19 and Fig. 20 , the first weighted maps of the three frames are all binary maps. Assume that the N frames of images originally acquired contain moving cars. For example, the white areas in the first weighted map are multiple moving cars. The first weighted map only shows the outlines of the cars. The area between the cars is not connected and feels fragmented. After morphological operations, that is, corrosion and expansion, the car area forms a connected domain, which is reflected in the pixel image as follows Fig.19 As shown, the area with a weight value of 0 surrounded by the area with a weight value of 1 is changed to a weight value of 1, thereby obtaining the final second weight map corresponding to the first weight map. Figure 1 The corresponding final second weight Figure 1 , with the first weight Figure 2 The corresponding final second weight Figure 2 , with the first weight Figure 3 The corresponding final second weight Figure 3 The present application connects the binary first weight map to obtain the second weight map, thereby avoiding the sense of fragmentation of the final fused image due to weight discreteness, making the fused image smoother and achieving better fusion effect.

[0106] See also Fig.21 In some embodiments, 059: acquiring N frames of ghost-free images according to N-1 second weight maps, a preset reference weight map, N-1 frames of registered images, and a reference image, including:

[0107] 0591: multiply each frame of the registered image by the corresponding second weight map to obtain an N-1 frame of de-ghosted image corresponding to the N-1 frame of the registered image; and

[0108] 0593: Multiply the reference image by the corresponding reference weight map to obtain a de-ghosted image corresponding to the reference image.

[0109] Please combine Figure 2 The generation module 15 is also used to: multiply each frame of the registered image by the corresponding second weight map to obtain the N-1 frame of de-ghosted image corresponding to the N-1 frame of the registered image; and multiply the reference image by the corresponding reference weight map to obtain the de-ghosted image corresponding to the reference image.

[0110] Please combine Figure 3 , one or more programs are executed by one or more processors 40, and the programs also include a method for executing the image processing methods in 0591 and 0593. That is, the processor 40 is also used to multiply each frame of the registered image by the corresponding second weight map to obtain the N-1 frame of the de-ghosted image corresponding to the N-1 frame of the registered image; and multiply the reference image by the corresponding reference weight map to obtain the de-ghosted image corresponding to the reference image.

[0111] See also Fig. 22 , is a schematic diagram of the principle of obtaining 4 ghost-free images based on 3 second weight maps, a preset reference weight map, 3 registered images and a reference image. Specifically, the registered image A' is multiplied by the corresponding second weight map Figure 1 , to obtain the ghost-free image 1 corresponding to the registered image A', and multiply the registered image B' by the corresponding second weight Figure 2 , to obtain the ghost-free image 2 corresponding to the registered image B', and multiply the registered image C' by the corresponding second weight Figure 3 , to obtain the ghost-free image 3 corresponding to the registered image C', and multiply the reference image P' by the corresponding preset reference weight map to obtain the ghost-free image 4 corresponding to the reference image P'. Among them, the registered image A', the registered image B', the registered image C' and the reference image P' are Fig. 22 The first column is composed of 4 frames arranged from top to bottom; the second column has weights Figure 1 , second weight Figure 2 , second weight Figure 3 The reference weight map is Fig. 22 The second column of the 4 frames are arranged from top to bottom, and the weight values ​​in the reference weight map can all be 1; ghost removal image 1, ghost removal image 2, ghost removal image 3 and ghost removal image 4 are Fig. 22 The third column of the 4 frames of images are arranged in order from top to bottom. In the ghost removal image 1, the pixel value of pixel ij is the pixel value of pixel ij in the registration image A' multiplied by the second weight Figure 1 The weight value of pixel ij in the de-ghosted image 2; the pixel value of pixel ij is the pixel value of pixel ij in the registered image B' multiplied by the second weight Figure 2 In the de-ghosting image 3, the pixel value of pixel ij is the pixel value of pixel ij in the registration image C' multiplied by the second weight Figure 3 in the de-ghosted image 4, the pixel value of pixel ij is the pixel value of pixel ij in the reference image P' multiplied by the weight value of pixel ij in the reference weight map; where i is the row number and j is the column number.

[0112] See also Fig.23In some embodiments, 07: fusing N frames of de-ghosted images to obtain a fused image, including:

[0113] 071: summing up the pixels at corresponding positions in the N-frame de-ghosting images to obtain a sum value map;

[0114] 073: Determine the normalization coefficients of the pixels at corresponding positions in the sum value map according to the N-1 second weight maps and the reference weight map; and

[0115] 075: The sum image is normalized according to the normalization system to obtain a fused image.

[0116] Please combine Figure 2 The fusion module 17 is also used for: summing the pixels at corresponding positions in the N frames of de-ghosted images to obtain a sum value map; determining the normalization coefficients of the pixels at corresponding positions in the sum value map according to the N-1 second weight maps and the reference weight map; and normalizing the sum value map according to the normalization system to obtain a frame of fused image.

[0117] Please combine Figure 3 , one or more programs are executed by one or more processors 40, and the programs also include the image processing methods for executing 071, 073 and 0757. That is, the processor 40 is also used to sum the pixels at corresponding positions in the N frames of de-ghosted images to obtain a sum value map; determine the normalization coefficients of the pixels at corresponding positions in the sum value map according to the N-1 second weight maps and the reference weight map; and perform normalization processing on the sum value map according to the normalization system to obtain a frame of fused image.

[0118] See also Fig.24 , is to sum up the pixels at corresponding positions in the four de-ghosted images to obtain the sum value map. The pixel value of pixel ij in the sum value map = the pixel value of pixel ij in de-ghosted image 1 + the pixel value of pixel ij in de-ghosted image 2 + the pixel value of pixel ij in de-ghosted image 3 + the pixel value of pixel ij in de-ghosted image 4; where i is the row number and j is the column number. For example, the pixel value of pixel 11 in the sum value map = the pixel value of pixel 11 in de-ghosted image 1 + the pixel value of pixel 11 in de-ghosted image 2 + the pixel value of pixel 11 in de-ghosted image 3 + the pixel value of pixel 11 in de-ghosted image 4, which is P'11. The pixel value of pixel 12 in the sum value map = the pixel value of pixel 12 in de-ghosted image 1 + the pixel value of pixel 12 in de-ghosted image 2 + the pixel value of pixel 12 in de-ghosted image 3 + the pixel value of pixel 12 in de-ghosted image 4, which is A'12+P'12. The pixel values ​​of other pixels in the sum value map are calculated using this method, and the results are as follows Fig.24 As shown, not listed here one by one.

[0119] See also Fig. 22 and Fig.25 In some embodiments, 073: determining the normalization coefficient of the pixels at corresponding positions in the sum value map according to the N-1 second weight maps and the reference weight map, including: taking the total number of final weight values ​​of the pixels at corresponding positions in the N-1 second weight maps and the reference weight map as 1 as the normalization coefficient of the pixels at corresponding positions in the sum value map.

[0120] For example, corresponding to the position of pixel 11, the second weight Figure 1 The weight value is 0, the second weight Figure 2 The weight value is 0, the second weight Figure 3 The weight value of the reference weight map is 0 and the weight value of the reference weight map is 1, then the normalization coefficient of pixel 11 is 1; corresponding to the position of pixel 12, the second weight Figure 1 The weight value is 1, the second weight Figure 2 The weight value is 0, the second weight Figure 3 The weight value of the reference weight map is 0 and the weight value of the reference weight map is 1, then the normalization coefficient of pixel 12 is 2; corresponding to the position of pixel 22, the second weight Figure 1 The weight value is 1, the second weight Figure 2 The weight value is 1, the second weight Figure 3 The weight value of the reference weight map is 1, and the normalization coefficient of pixel 22 is 4; corresponding to the position of pixel 24, the second weight Figure 1 The weight value is 1, the second weight Figure 2 The weight value is 1, the second weight Figure 3 If the weight value of is 0 and the weight value of the reference weight map is 1, the normalization coefficient of the pixel 22 is 3. The normalization coefficients of the pixels at other positions in the sum value map are also calculated and obtained according to this method, which are not listed here one by one.

[0121] Please continue reading Fig.25 , the sum value map is normalized according to the normalization system to obtain a frame of fused image. Specifically, the pixel value of each pixel in the sum value map is divided by the normalization coefficient of the corresponding pixel position. Corresponding to the position of pixel 11, the pixel value of the fused image = P'11; corresponding to the position of pixel 12, the pixel value of the fused image = (A'12+P'12) / 2; corresponding to the position of pixel 22, the pixel value of the fused image = (A'22+B'22+C'22+P'22) / 4; corresponding to the position of pixel 24, the pixel value of the fused image = (A'24+B'24+P'24) / 3. The pixel values ​​of other pixels of the fused image are calculated and obtained by this method, which are not listed here one by one.

[0122] See also Fig.26, which is a comparison chart of the fused image obtained by the image processing method of the present application and the fused image obtained by the common multi-frame fusion algorithm. It is obvious from the figure that the fused image obtained by the image processing method of the present application is clearer, has more details, and has a better visual effect. Due to the introduction of noise estimation and weight optimization, the details of the fused image can be more highly retained in the moving area, so the license plate number and the texture around the tire are richer and clearer.

[0123] See also Fig. 27 The present application also provides a non-volatile computer-readable storage medium 200 storing a computer program. When the computer program 202 is executed by one or more processors 40, the image processing method of any embodiment described above is implemented.

[0124] For example, when the program 202 is executed by the processor 40, the following image processing method is implemented:

[0125] 01: Get N frames of images, where N ≥ 2;

[0126] 03: Determine a reference image among N frames of images, and the remaining N-1 frames of images are images to be processed;

[0127] 05: Generate N frames of ghost-free images based on the reference image and the image to be processed; and

[0128] 07: Perform fusion processing on N frames of de-ghosted images to obtain a fused image.

[0129] For another example, when the program 202 is executed by the processor 40, the following image processing method is implemented:

[0130] 051: Register each frame of the image to be processed with the reference image to determine N-1 frames of registered images;

[0131] 053: Perform noise estimation on each frame of the registered image and determine N-1 noise levels;

[0132] 055: Perform motion detection on each frame of the registered image to determine N-1 first weight maps;

[0133] 057: Perform morphological operations on each first weight map to determine N-1 second weight maps; and

[0134] 059: Obtain N frames of de-ghosted images according to N-1 second weight maps, a preset reference weight map, N-1 frames of registered images, and a reference image.

[0135] For another example, when program 202 is executed by processor 40, it can also implement the image processing methods in 0511, 0513, 0514, 0515, 0517, 0531, 0533, 0535, 05351, 05353, 05355, 0551, 0553, 0555, 0591, 0593, 071, 073 and 0757.

[0136] In the non-volatile computer-readable storage medium 200 in the present application, each frame of the image to be processed is first registered with the reference image to obtain N-1 frames of registered images, and then the N-1 frames of registered images and the reference images are used to perform noise estimation, motion detection, morphological operations, etc., so that the misalignment and superposition of the image caused by camera movement or object movement during noise estimation, motion detection, and morphological operations are minimized, thereby ensuring the accuracy of noise estimation, motion detection, morphological operations, etc. and the best results.

[0137] In the description of this specification, the descriptions with reference to the terms "certain embodiments", "in an example", "exemplarily", etc., mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0138] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0139] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An image processing method, characterized in that: include: Obtain N frames of images, where N ≥ 2; Determine a reference image frame from the N frames of images, and the remaining N-1 frames of images are images to be processed; Registering each frame of the image to be processed with the reference image to determine N-1 frames of registered images; Perform noise estimation on each frame of the registered image to determine N-1 noise levels; the N-1 noise levels are obtained based on the N-1 frame noise difference map and the number of frames N, and the N-1 frame noise difference map is obtained based on the noise difference value between the reference sampling map corresponding to the reference image and the registration sampling map corresponding to each frame of the registered image at the corresponding pixel position; Obtaining a pixel difference map of frame N-1 according to the pixel difference between the reference image and the corresponding pixel position of each frame of the registered image; Amplifying the pixel difference in the pixel difference map of the corresponding frame according to the noise level to obtain an N-1 frame noise feature map; Performing threshold detection on the characteristic values ​​of all pixels in the noise characteristic map of each frame, and setting an initial weight value for each pixel to obtain N-1 first weight maps; Performing a morphological operation on each of the first weight maps to determine N-1 second weight maps; Acquire N frames of ghost-free images according to N-1 of the second weight maps, a preset reference weight map, N-1 frames of the registered images, and the reference image; and The N frames of de-ghosted images are fused to obtain a frame of fused image.

2. The image processing method according to claim 1, characterized in that: The registering each frame of the image to be processed with the reference image to determine N-1 frames of registered images includes: Extracting feature points from each frame of the image to be processed and the reference image to obtain a plurality of feature points, and describing the features of each feature point; Matching the feature points of each frame of the image to be processed with the feature points of the reference image to obtain a plurality of feature point pairs; Solving the transformation matrix between each frame of the image to be processed and the reference image in the matched feature point pairs; and The corresponding image to be processed is registered and aligned with the reference image according to the transformation matrix to obtain N-1 frames of registered images.

3. The image processing method according to claim 1, characterized in that The performing noise estimation on each frame of the registered image to determine N-1 noise levels includes: Taking M*M pixels as a unit, noise sampling is performed on the reference image and the N-1 frame of the registered image to obtain a reference sampling image corresponding to the reference image and an N-1 frame of the registered image corresponding to each other, 2≤M≤15% of the total number of pixels in the length direction; Obtaining a noise difference map of frame N-1 according to the noise difference value between the reference sampling map and the corresponding pixel position of each frame of the registration sampling map; and The N-1 noise levels are obtained according to the noise difference map of the N-1 frame and the frame number N.

4. The image processing method according to claim 3, characterized in that: The obtaining the N-1 noise levels according to the noise difference map of the N-1 frame and the frame number N includes: Calculating the overall difference value of the noise difference map of each frame to obtain N-1 overall difference values; Obtaining the median of the N-1 overall difference values; and The ratio of the median to the number of frames N is used as the noise level.

5. The image processing method according to claim 1, characterized in that: The step of acquiring the N frames of de-ghosted images according to the N-1 second weight maps, a preset reference weight map, the N-1 frames of the registered images, and the reference image comprises: Multiplying each frame of the registered image by the corresponding second weight map to obtain an N-1 frame of de-ghosted image corresponding to the N-1 frame of the registered image; and The reference image is multiplied by the corresponding reference weight map to obtain the de-ghosted image corresponding to the reference image.

6. The image processing method according to claim 5, characterized in that: The fusing process of the N frames of de-ghosted images to obtain a frame of fused image includes: Performing sum processing on pixels at corresponding positions in the N frames of de-ghosted images to obtain a sum value graph; Determine normalization coefficients of pixels at corresponding positions in the sum map according to N-1 of the second weight maps and the reference weight map; The sum value map is normalized according to the normalization system to obtain a frame of the fused image.

7. An image processing device, characterized in that: include: An acquisition module, used for acquiring N frames of images, where N ≥ 2; A determination module, used to determine a reference image from the N frames of images, and the remaining N-1 frames of images are images to be processed; The generating module is used to register each frame of the image to be processed with the reference image to determine N-1 frames of registered images; perform noise estimation on each frame of the registered image to determine N-1 noise levels; the N-1 noise levels are obtained according to the N-1 frame noise difference map and the number of frames N, and the N-1 frame noise difference map is obtained according to the noise difference value between the reference sampling map corresponding to the reference image and the registered sampling map corresponding to each frame of the registered image at the corresponding pixel positions; obtain the pixel difference between the reference image and the corresponding pixel position of each frame of the registered image according to the pixel difference between the reference image and the corresponding pixel position of each frame of the registered image. Take N-1 frames of pixel difference maps; amplify the pixel differences in the pixel difference maps of the corresponding frames according to the noise level to obtain N-1 frames of noise feature maps; perform threshold detection on the characteristic values ​​of all pixels in the noise feature maps of each frame, and set an initial weight value for each pixel to obtain N-1 first weight maps; perform morphological operations on each of the first weight maps to determine N-1 second weight maps; obtain N frames of ghost-free images according to the N-1 second weight maps, the preset reference weight map, the N-1 frames of the registered images, and the reference image; and The fusion module is used to perform fusion processing on the N frames of de-ghosted images to obtain a frame of fused image.

8. The image processing device according to claim 7, characterized in that: The generating module is further used to extract feature points from each frame of the image to be processed and the reference image to obtain a plurality of feature points, and to describe the features of each feature point; match the feature points of each frame of the image to be processed with the feature points of the reference image to obtain a plurality of feature point pairs; Solving the transformation matrix between each frame of the image to be processed and the reference image in the feature point pairs obtained by matching; And aligning the corresponding image to be processed with the reference image according to the transformation matrix to obtain N-1 frames of registered images.

9. A terminal, characterized in that: include: One or more processors, memory; and One or more programs, wherein one or more of the programs are stored in the memory and executed by one or more of the processors, and the programs include instructions for executing the image processing method according to any one of claims 1 to 6.

10. A non-volatile computer-readable storage medium storing a computer program, which, when executed by one or more processors, implements the image processing method according to any one of claims 1 to 6.

Images