Image processing method, device, electronic device and readable storage medium
By acquiring and registering motion mask images of multi-frame images, detecting and eliminating ghosts in night scene shooting, the problem of low image clarity in the prior art is solved, and higher image clarity is achieved.
Patent Information
- Application Number
- CN202010616014.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-29
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-06-29
AI Technical Summary
The prior art image noise problem caused by insufficient light during night scene shooting results in low detection accuracy in motion areas, resulting in ghosting in the synthetic image, affecting the image clarity.
By acquiring the reference image at the reference exposure compensation value and the fusion image at the multiple control exposure compensation values, the motion mask image between each control exposure compensation value is determined and registered to detect the ghost area, thereby performing Image fusion to eliminate ghosting and improve image clarity.
Improves the accuracy of ghost area detection, eliminates ghosting in the image, and obtains images with higher definition.
Smart Images

Figure CN113870300B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and readable storage medium. Background Art
[0002] With the development of intelligent terminal technology, the use of mobile terminal devices (such as smart phones, tablet computers, etc.) is becoming more and more popular. Most mobile terminal devices have built-in cameras, and with the enhancement of mobile terminal processing capabilities and the development of camera technology, the quality of captured images is getting higher and higher. In daily life, more and more users use mobile terminal devices such as smart phones and tablet computers to capture images, and the requirements for photo quality are getting higher and higher.
[0003] However, due to the limitation of shooting scenes, especially in special scenes such as night scenes, since the light at night is relatively dim, noise may be introduced when shooting images, resulting in unclear images. Therefore, it is necessary to perform noise reduction on the captured images. However, in the prior art, when performing multi-frame noise reduction, two frames of original images are usually compared to distinguish between moving areas and non-moving areas. However, in this method, the moving areas and non-moving areas are not completely distinguished, resulting in low accuracy of moving area detection. In this way, the final synthesized image still has ghosts, resulting in unclear images. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and readable storage medium, so as to improve the problem in the prior art that the motion area detection accuracy is low, resulting in unclear synthesized images.
[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:
[0006] Obtaining a reference image at a reference exposure compensation value and fused images at N comparison exposure compensation values, wherein the reference image is obtained by fusing at least two frames of images captured at the reference exposure compensation value; and the fused image is obtained by fusing at least two frames of images captured at the comparison exposure compensation value;
[0007] Determine a motion mask image between at least two frames of images collected at each comparison exposure compensation value, wherein pixel values of pixels in the motion mask image are used to characterize the motion amplitude of corresponding pixels in at least two frames of images collected at each comparison exposure compensation value;
[0008] The fused image under each contrast exposure compensation value and the motion mask image corresponding to each contrast exposure compensation value are respectively registered with the reference image to obtain a registered image corresponding to each contrast exposure compensation value and a registered motion mask image;
[0009] Determine a ghost area in the registered image at each comparison exposure compensation value based on the registered image at each comparison exposure compensation value, the reference image, and the registered motion mask image corresponding to each comparison exposure compensation value;
[0010] Based on the ghost area, the registration images corresponding to the N comparison exposure compensation values and the reference image are fused to obtain a target image.
[0011] In the above implementation process, by obtaining a motion mask image between at least two frames of images captured for each control exposure compensation value, and then registering the motion mask image, the ghost area in the registered image under each control exposure compensation value can be accurately detected in combination with the registered motion mask image, thereby improving the accuracy of ghost area detection. In this way, when image fusion is performed based on the ghost area, the ghost can be eliminated well and a higher definition image can be obtained.
[0012] Optionally, fusing the registration images corresponding to the N control exposure compensation values and the reference image based on the ghost area to obtain the target image includes:
[0013] Selecting two images as to-be-fused images from N+1 images formed by the reference image and the N registered images and an intermediate fused image formed by fusing two images from the N+1 images;
[0014] According to the highlight area and the non-ghost area in the ghost area of one image to be fused, the image to be fused is fused with another image to be fused, and the target image is obtained after all the images in the N+1 images are fused once;
[0015] Among them, when the image to be fused is an intermediate fused image, its ghost area is the area corresponding to the ghost area of the background image in the two images participating in the fusion, and its highlight area is the area corresponding to the highlight area in the background image.
[0016] In the above implementation process, by fusing the highlight area in the ghost area in the image, the highlight area pixels in the images with different brightness can be better fused, so that the clarity of the fused image is higher.
[0017] Optionally, the fusing the image to be fused with another image to be fused according to a highlight area and a non-ghost area in a ghost area of the image to be fused, comprises:
[0018] Taking the pixel points of the highlight area in the one image to be fused and the pixel points of the area corresponding to the non-highlight area in the ghost area of the one image to be fused in the other image to be fused as the pixel points of the corresponding area in the image obtained after the two images to be fused;
[0019] Fusing pixel points of other regions except the ghost region in the one image to be fused with pixel points of a region corresponding to the other region in the other image to be fused;
[0020] The exposure compensation value corresponding to the ghost area in the one image to be fused is smaller than the exposure compensation value corresponding to the ghost area in the other image to be fused.
[0021] In the above implementation process, by fusing in the above manner, the pixels in the highlight areas of images with different brightness can be well fused, so that the clarity of the fused image is higher.
[0022] Optionally, fusing pixel points of other regions except the ghost region in the one image to be fused with pixel points of a region corresponding to the other region in the other image to be fused includes:
[0023] Selecting one of the image to be fused and the other image to be fused as a background image, and the other image as a non-background image;
[0024] Determine the pixel size of each corresponding pixel point in other areas of the substrate image except the ghost area;
[0025] Determine, based on the pixel size, a fusion weight of each corresponding pixel point in the other area in the substrate image and a fusion weight of each corresponding pixel point in the area corresponding to the other area in the non-substrate image;
[0026] Based on the fusion weights of each corresponding pixel point in the other areas in the background image and the fusion weights of each corresponding pixel point in the area corresponding to the other areas in the non-background image, the other areas in the background image and the corresponding pixels in the area corresponding to the other areas in the non-background image are fused.
[0027] In the above implementation process, pixel fusion is performed based on the fusion weight, which can effectively extract the highlight pixels in the image and achieve a good fusion of the highlight pixels and the low-light pixels.
[0028] Optionally, the image whose order in which the N+1 images are sorted in descending order or in ascending order according to the corresponding exposure compensation values is i is the i-th image, and the selecting two images from the N+1 images formed by the reference image and the N registered images and the intermediate fused image formed by fusing two images of the N+1 images as the images to be fused includes:
[0029] Fusing the first image with the second image to form an intermediate fused image;
[0030] i is sequentially selected from 3 to N+1, and an intermediate fused image formed by fusion of all images before the i-th image in the order and the i-th image are selected as images to be fused.
[0031] In the above implementation process, the images are fused in sequence according to the size of the exposure compensation values, so that images with different brightness can be better fused.
[0032] Optionally, the image whose order in which the N+1 images are sorted in descending order or in ascending order according to the corresponding exposure compensation values is i is the i-th image, and the selecting two images from the N+1 images formed by the reference image and the N registered images and the intermediate fused image formed by fusing two images of the N+1 images as the images to be fused includes:
[0033] i is taken as an odd number from 1 to N+1, and the adjacent i-th image and i+1-th image in the N+1 images are fused in sequence according to the sorting order to form an intermediate fused image;
[0034] According to the order in which the intermediate fusion images are obtained, two adjacent intermediate fusion images are selected in sequence as the images to be fused.
[0035] In the above implementation process, adjacent images with different brightness are fused, so as to reduce the brightness error during fusion and achieve better fusion effect.
[0036] Optionally, when the image to be fused is one of the N+1 images, the ghost area in the corresponding image to be fused is determined in the following manner:
[0037] Binarize the initial ghost area in the corresponding image to be fused to obtain a binary ghost area;
[0038] The binary ghost region is divided to obtain a plurality of non-adjacent ghost regions.
[0039] In the above implementation process, binarization is performed on the ghost area, which can effectively filter out the noise in the ghost area.
[0040] Optionally, the highlight area in the ghost area of the corresponding image to be fused is determined in the following manner:
[0041] Determine the average pixel value of each pixel in the ghost area of the corresponding image to be fused;
[0042] If the average pixel value of the corresponding ghost area is greater than a preset value, it is determined that the corresponding ghost area in the corresponding image to be fused is a highlight area.
[0043] In the above implementation process, by dividing the ghost area, the highlight area in the ghost area can be found more accurately.
[0044] Optionally, when the image to be fused is the registered image, the initial ghost area in the corresponding registered image is determined by:
[0045] Based on the pixel deviation between the registered image at each comparison exposure compensation value and the reference image, and the pixel value in the registered motion mask image corresponding to each comparison exposure compensation value, an initial ghost area in the registered image at each comparison exposure compensation value is determined.
[0046] Optionally, determining the initial ghost area in the registered image at each comparison exposure compensation value based on the pixel deviation between the registered image at each comparison exposure compensation value and the reference image, and the pixel value in the registered motion mask image corresponding to each comparison exposure compensation value, comprises:
[0047] Determine the pixel difference of each corresponding pixel point between the registration image and the reference image at each comparison exposure compensation value;
[0048] Determine whether the pixel value difference of the corresponding pixel point is greater than a first threshold, and whether the pixel value of the corresponding pixel point in the registered motion mask image corresponding to each comparison exposure compensation value is greater than a second threshold;
[0049] If the pixel value difference of the corresponding pixel point is greater than the first threshold and the pixel value of the corresponding pixel point in the registered motion mask image is greater than the second threshold, the position of the corresponding pixel point in the registered image under the corresponding comparison exposure compensation value is determined to be the initial ghost area.
[0050] In the above implementation process, ghost detection in the registered image is performed in combination with the pixel values of the pixel points in the registered motion mask image. This can avoid the problem in the prior art of performing ghost detection only through the pixel difference between the two original images, resulting in a large number of highlight areas being mistakenly detected as ghost areas, thereby effectively improving the accuracy of ghost detection.
[0051] Optionally, the registering the fused image at each comparison exposure compensation value and the motion mask image corresponding to each comparison exposure compensation value with the reference image to obtain the registered image corresponding to each comparison exposure compensation value and the registered motion mask image includes:
[0052] Adjusting the image brightness of the fused image at each control exposure compensation value to the image brightness of the reference image;
[0053] Determine registration reference data for registering the fused image at each comparison exposure compensation value after the image brightness is adjusted with the reference image;
[0054] The fused image at each comparison exposure compensation value is registered based on the registration reference data to obtain a registered image corresponding to each comparison exposure compensation value.
[0055] In the above implementation process, by registering the fused image corresponding to each comparison exposure compensation value to the reference image, subsequent ghost detection and image fusion can be facilitated.
[0056] Optionally, the registering the fused image at each comparison exposure compensation value and the motion mask image corresponding to each comparison exposure compensation value with the reference image to obtain the registered image corresponding to each comparison exposure compensation value and the registered motion mask image includes:
[0057] The motion mask image corresponding to each comparison exposure compensation value is registered based on the registration reference data to obtain a registered motion mask image corresponding to each comparison exposure compensation value.
[0058] In the above implementation process, the motion mask image is registered based on the registration reference data, so that when detecting ghosts, the pixel values at corresponding positions in the registered motion mask image can be effectively combined for detection.
[0059] Optionally, after obtaining the target image, the method further includes:
[0060] The target image is adjusted by using a contrast-limited adaptive histogram equalization algorithm, so that an image with good contrast and clarity can be obtained.
[0061] Optionally, obtaining a reference image at a reference exposure compensation value and a fused image at N comparison exposure compensation values includes:
[0062] Acquire at least two frames of images captured at each of the N reference exposure compensation values and at least two frames of images captured at the reference exposure compensation value for the same scene;
[0063] At least two frames of images captured at each control exposure compensation value are fused to obtain fused images at the N control exposure compensation values, and at least two frames of images captured at the reference exposure compensation value are fused to obtain a reference image at the reference exposure compensation value.
[0064] In the above implementation process, by fusing the collected images, the noise in the collected images can be effectively filtered out, thereby achieving the effect of image denoising.
[0065] Optionally, at least two frames of images are fused in the following manner:
[0066] Determine the image with the highest definition among the at least two frames of images as a reference image;
[0067] The other frame images of the at least two frame images are fused with the reference image to obtain a corresponding fused image.
[0068] In the above implementation process, the image with the highest definition is selected as the reference image, so that after fusion, a well-denoised fused image can be obtained.
[0069] Optionally, fusing other frame images of the at least two frame images into the reference image to obtain a corresponding fused image includes:
[0070] Determine the pixel difference between each corresponding pixel in each other frame image and the reference image;
[0071] Determine the fusion weight of each corresponding pixel point in each other frame image based on the pixel difference;
[0072] Based on the fusion weights of the corresponding pixels in each of the other frame images, the other frame images are fused with the corresponding pixels in the reference image to obtain a corresponding fused image.
[0073] In the above implementation process, pixel fusion is performed based on the fusion weight, which can effectively extract the highlight pixels in the image and achieve a good fusion of the highlight pixels and the low-light pixels.
[0074] Optionally, each of the at least two frames of images collected for comparison with the exposure compensation value includes a reference image, and determining a motion mask image between each of the at least two frames of images collected for comparison with the exposure compensation value includes:
[0075] Obtaining pixel differences of corresponding pixel points between other frame images in at least two frame images collected by comparing the exposure compensation value and the reference image;
[0076] Using the pixel difference of each corresponding pixel point as the initial motion mask image between each other frame image and the reference image, and obtaining a total of multiple initial motion mask images;
[0077] The multiple initial motion mask images are superimposed to obtain a motion mask image between at least two frames of images collected according to each comparison exposure compensation value.
[0078] In the above implementation process, the corresponding motion mask image is determined by obtaining the pixel difference between at least two frames collected according to each contrast exposure compensation value, so that the motion area in the image can be effectively determined.
[0079] Optionally, the number of images captured at each control exposure compensation value and the number of images captured at the reference exposure compensation value are determined according to the sensitivities corresponding to different exposure compensation values.
[0080] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:
[0081] An image acquisition module, used to obtain a reference image at a reference exposure compensation value and fused images at N comparison exposure compensation values, wherein the reference image is obtained by fusion of at least two frames of images collected at the reference exposure compensation value; and the fused image is obtained by fusion of at least two frames of images collected at the comparison exposure compensation value;
[0082] A motion mask image acquisition module, used to determine a motion mask image between at least two frames of images collected at each comparison exposure compensation value, wherein the pixel values of the pixels in the motion mask image are used to represent the motion amplitude of the corresponding pixel in at least two frames of images collected at each comparison exposure compensation value;
[0083] An image registration module, used to register the fused image under each contrast exposure compensation value and the motion mask image corresponding to each contrast exposure compensation value with the reference image, respectively, to obtain a registered image corresponding to each contrast exposure compensation value and a registered motion mask image;
[0084] a ghost region detection module, configured to determine a ghost region in the registered image at each comparison exposure compensation value based on the registered image at each comparison exposure compensation value, the reference image, and the registered motion mask image corresponding to each comparison exposure compensation value;
[0085] An image fusion module is used to fuse the registration images corresponding to the N control exposure compensation values and the reference image based on the ghost area to obtain a target image.
[0086] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect are performed.
[0087] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the method provided in the first aspect are performed.
[0088] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0090] Figure 1 A schematic diagram of the structure of an electronic device for executing an image processing method provided in an embodiment of the present application;
[0091] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application;
[0092] Figure 3 A schematic diagram of fusing at least two frames of images under a comparison of exposure compensation values provided in an embodiment of the present application;
[0093] Figure 4 A schematic diagram of obtaining a motion mask image between at least two frames of images under a comparison exposure compensation value provided by an embodiment of the present application;
[0094] Figure 5 A schematic diagram of fusing images to be fused provided in an embodiment of the present application;
[0095] Figure 6 A structural block diagram of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0096] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application.
[0097] An image processing method provided in an embodiment of the present application obtains a motion mask image between at least two frames of images captured at each control exposure compensation value, and then aligns the motion mask image, so that the ghost area in the registered image under each control exposure compensation value can be accurately detected in combination with the registered motion mask image, thereby improving the accuracy of ghost area detection. In this way, when image fusion is performed based on the ghost area, the ghost can be eliminated well to obtain an image with higher clarity.
[0098] Please refer to Figure 1 , Figure 1 A structural diagram of an electronic device for executing an image processing method provided in an embodiment of the present application, the electronic device may include: at least one processor 110, such as a CPU, at least one communication interface 120, at least one memory 130 and at least one communication bus 140. Among them, the communication bus 140 is used to realize direct connection and communication between these components. Among them, the communication interface 120 of the device in the embodiment of the present application is used to communicate signaling or data with other node devices. The memory 130 can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 130 can optionally also be at least one storage device located away from the aforementioned processor. The memory 130 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 110, the electronic device executes the following Figure 2 In the method process shown, for example, the memory 130 can be used to store images corresponding to various exposure compensation values. When the processor 110 performs image processing, it can obtain the corresponding image from the memory 130 to perform corresponding processing to obtain a fused target image.
[0099] In the embodiment of the present application, the electronic device may be a terminal device. When the electronic device is a terminal device, the terminal device may also include a camera, which may be triggered to turn on by an application program instruction to realize a photo or video function, and the camera may send the collected image to a processor in the terminal device for corresponding processing. The terminal device may be a smart phone, a tablet computer, a personal digital assistant, a wearable device, or other hardware device with various operating systems and imaging devices.
[0100] Understandably, Figure 1 The structure shown is for illustration only, and the electronic device may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0101] Please refer to Figure 2 , Figure 2 A flowchart of an image processing method provided in an embodiment of the present application, the method comprising the following steps:
[0102] Step S110: obtaining a reference image at a reference exposure compensation value and fused images at N comparison exposure compensation values.
[0103] Among them, exposure compensation is a way to control exposure. It is the parameters of the shutter and aperture combination obtained after the terminal device measures the light of the subject, and then uses exposure compensation to artificially change the shutter speed obtained by this measurement.
[0104] The exposure compensation value generally ranges from -3 to +3EV. For example, the exposure compensation value can be EV+2, EV+1, EV0, EV-1, EV-2. If the ambient light is dim, the exposure value can be increased to highlight the clarity of the picture. Of course, in actual applications, the exposure compensation values corresponding to different terminal devices can be more or less. When taking images, you can select the corresponding exposure compensation value to take the image according to your needs.
[0105] However, due to the influence of factors such as the light intensity in the shooting scene and the degree of jitter of the terminal device, the terminal device may produce unclear images or introduce noise when shooting images. Therefore, it is generally possible to use the corresponding exposure compensation value to shoot multiple frames of original images, and then select clear original images from them for fusion noise reduction processing.
[0106] In the embodiment of the present application, in order to obtain an image with better clarity, multiple exposure compensation values may be used for image capture, that is, a base exposure compensation value and N control exposure compensation values are used to capture the image, and N may be an integer greater than or equal to 1. The base exposure compensation value may be understood as a normal exposure compensation value, and the change in exposure compensation may be an increase or decrease in the exposure multiple based on the normal exposure compensation value. For example, the control exposure compensation value may be a change based on the base exposure compensation value. For example, the base exposure compensation value is EV0, and the N control exposure compensation values may include EV+2, EV+1, EV-1, and EV-2, wherein EV+2 may be understood as an increase of two times the exposure based on EV0, and EV+1 may be understood as an increase of one times the exposure based on EV0. Similarly, EV-1 may be understood as a decrease of one times the exposure based on EV0, and EV-2 may be understood as a decrease of two times the exposure based on EV0.
[0107] It should be noted that, in actual situations, the specific values of the baseline exposure compensation value and the N control exposure compensation values can be set according to needs, or appropriate baseline exposure compensation values and N control exposure compensation values can be selected according to needs. In the embodiment of the present application, for the convenience of description, in the following description process, the above-mentioned baseline exposure compensation value is EV0, and the N control exposure compensation values are EV+2, EV+1, EV-1, and EV-2 as examples.
[0108] In order to better realize image denoising and achieve better results, after collecting multiple frames of images at different exposure compensation values, the multiple frames of images can be fused and denoised to obtain corresponding fused images. Among them, at the reference exposure compensation value, one frame or at least two frames of images can be collected. When collecting at least two frames of images, the reference image is obtained by fusion of at least two frames of images collected at the reference exposure compensation value. For example, the reference image corresponding to EV0 is obtained by fusion of at least two frames of EV0 images collected. The fused image at the control exposure compensation value is also obtained by fusion of at least two frames of images collected at the control exposure compensation value. For example, the fused image corresponding to EV+1 is obtained by fusion of at least two frames of EV+1 images collected. The reference image and fused image obtained in this way can be an image with good denoising effect, which is conducive to improving the accuracy of ghost detection in the future.
[0109] It should be noted that if only one frame of image is collected under the reference exposure compensation value, the reference image is the collected frame of image. If at least two frames of image are collected under the reference exposure compensation value, the reference image is the fused image after the fusion processing of the at least two frames of image.
[0110] Step S120: determining a motion mask image between at least two frames of images collected at each comparison exposure compensation value.
[0111] The pixel value of the pixel in the motion mask image can be used to characterize the motion amplitude of the corresponding pixel in at least two frames of images collected by comparing the exposure compensation value. It can be understood that due to the jitter during shooting and other reasons, the object in the at least two frames of images obtained by shooting has motion offset, so if the motion offset is large, it will be reflected in the motion mask image as a large pixel value of the pixel, that is, it indicates that the motion amplitude of the corresponding pixel in at least two frames of images is large.
[0112] Step S130: respectively registering the fused image at each contrast exposure compensation value and the motion mask image corresponding to each contrast exposure compensation value with the reference image to obtain a registered image corresponding to each contrast exposure compensation value and a registered motion mask image.
[0113] Since the images acquired under different exposure compensation values may be different, in order to facilitate subsequent image fusion, the fused image and the motion mask image need to be registered. For example, with the reference image as a reference, the fused image under each control exposure compensation value is registered with the reference image, so that each pixel in each fused image is aligned with the corresponding pixel in the reference image, and the registered image corresponding to each control exposure compensation value is obtained. And each motion mask image is registered to the reference image, so that each pixel in each motion mask image is aligned with the corresponding pixel in the reference image, and the registered motion mask image corresponding to each control exposure compensation value is obtained, which can facilitate the subsequent detection of ghost areas in the registered image.
[0114] Step S140: determining a ghost area in the registered image at each comparison exposure compensation value based on the registered image at each comparison exposure compensation value, the reference image, and the registered motion mask image corresponding to each comparison exposure compensation value.
[0115] During the shooting process, due to the movement of objects in the shooting scene or the shaking caused by shooting, some areas in the obtained image appear blurred or translucent. This phenomenon is called ghosting, and the area where these phenomena appear in the image is called the ghost area. In order to obtain a clear image, the ghost area in the image needs to be detected during the image fusion process so that the ghost area in the image can be eliminated during the image fusion process, making the obtained fused image clearer.
[0116] Since the highlight areas in the images obtained with different exposure compensation values are obviously different, if only the pixel difference between the images obtained with different exposure compensation values is used as the basis for ghost detection, a large number of highlight areas in the image may be mistakenly detected as ghost areas, resulting in low detection accuracy of the ghost areas, so that the fused image finally obtained still has the problem of being unclear. Therefore, in the embodiment of the present application, in order to improve the accuracy of ghost area detection, when determining the ghost area in the registered image, it is also necessary to combine the registered motion mask image for detection.
[0117] Step S150: Based on the ghost area, the registered images corresponding to the N control exposure compensation values and the reference image are fused to obtain a target image.
[0118] After detecting the ghost area in each registered image, the N registered images and the reference image, a total of N+1 images, can be fused based on the ghost area, so that the ghost area in the image can be eliminated during the fusion process to obtain a target image with higher definition.
[0119] In the above implementation process, by obtaining a motion mask image between at least two frames of images captured for each control exposure compensation value, and then registering the motion mask image, the ghost area in the registered image under each control exposure compensation value can be accurately detected in combination with the registered motion mask image, thereby improving the accuracy of ghost area detection. In this way, when image fusion is performed based on the ghost area, the ghost can be eliminated well and a higher definition image can be obtained.
[0120] As an implementation manner, in order to perform noise reduction processing on the images captured at each exposure compensation value, the images captured at each exposure compensation value may be fused first. During the fusion process, at least two frames of images captured at each of N reference exposure compensation values and at least two frames of images captured at a reference exposure compensation value for the same scene may be first obtained. Then, the at least two frames of images captured at each reference exposure compensation value may be fused to obtain fused images at the N reference exposure compensation values. Furthermore, the at least two frames of images captured at the reference exposure compensation value may be fused to obtain a reference image at the reference exposure compensation value.
[0121] Among them, the same scene can refer to the image acquisition of the same object in the scene at each exposure compensation value, such as taking a picture of the moon, a person or an animal, etc. When performing image acquisition, the terminal device can pre-store N comparison exposure compensation values and reference exposure compensation values, that is, the specific values of the above-mentioned exposure compensation values of EV+2, EV+1, EV0, EV-1, and EV-2 can be pre-set in the terminal device, so that after receiving the user-triggered photo command, the terminal device can automatically set the photo parameters to the corresponding exposure compensation value for image acquisition. For example, when the terminal device sets the exposure compensation value to EV+2, it acquires at least two corresponding frames of images for the scene. After acquiring the image, the exposure compensation value is adjusted to EV+1, and then the image acquisition is continued, so that the image at each exposure compensation value can be acquired.
[0122] Among them, the at least two frames of images captured by the terminal device at each exposure compensation value can be obtained by continuously shooting the same scene in a short time (such as 1S), and the at least two frames of images captured can form a group of image frame sequences according to the chronological order of capture time.
[0123] In addition, when performing image acquisition, image acquisition for the same scene can be performed using one camera, or using two or more cameras, and then the images acquired by multiple cameras can be used as images acquired under corresponding exposure compensation values.
[0124] In the above implementation process, by fusing the collected images, the noise in the collected images can be effectively filtered out, thereby achieving the effect of image denoising.
[0125] As an implementation mode, in the above embodiment, when capturing images at different exposure compensation values, in order to facilitate image capture, the conditions for the terminal device to capture images can be set in advance, that is, before image capture, it can also be determined whether the ambient brightness of the shooting scene is less than a preset brightness threshold. If so, at least two frames of images captured at each of the N comparison exposure compensation values for the same scene and at least one frame of image captured at the baseline exposure compensation value are obtained.
[0126] Among them, an ambient light sensor can be set in the terminal device, and the ambient light sensor can be used to obtain the ambient brightness of the shooting scene, and the preset brightness threshold can be flexibly set according to actual needs. In this implementation, if the terminal device receives a photo command, it will only use different exposure compensation values to collect images when the ambient brightness is less than the preset brightness threshold, indicating that the ambient brightness is dark at this time. In order to obtain a better image, it is necessary to collect images under different exposure compensation values, and then these collected images can be processed accordingly to obtain a better image.
[0127] As another implementation, before image acquisition, a photographing instruction may be received first, and then it is determined whether the photographing mode corresponding to the photographing instruction is a night scene photographing mode. If so, at least two frames of images acquired at each of the N comparative exposure compensation values and at least one frame of image acquired at the reference exposure compensation value for the same scene are acquired. That is to say, in this case, image acquisition at different exposure compensation values is adopted only when in the night scene photographing mode, indicating that when performing night scene image acquisition at this time, in order to obtain images with better effects, it is necessary to acquire images at different exposure compensation values, and then these acquired images are processed accordingly to obtain images with better effects.
[0128] Of course, the above two methods can also be combined, that is, before image acquisition, it can be determined whether the ambient brightness of the shooting scene is less than the preset brightness threshold and whether it is in the night scene shooting mode. When the ambient brightness is less than the preset brightness threshold and is in the night scene shooting mode, different exposure compensation values will be used for image acquisition.
[0129] It can be understood that in actual application, the judgment conditions for image acquisition can be flexibly set according to actual needs. It is not limited to the above-mentioned several implementation methods. Other judgment conditions can also be set. For example, after receiving the photo command, it can be determined whether it is in the night scene shooting mode and whether the terminal device is in a stable state. When it is in the night scene shooting mode and the terminal device is in a stable state, different exposure compensation values are used for image acquisition. Other judgment conditions are not listed here one by one.
[0130] As an implementation method, in order to facilitate the terminal device to automatically capture images at different exposure compensation values, the number of images taken at different exposure compensation values can be pre-set, such as setting the number of images taken at EV+1 to 5 frames, the number of images taken at EV0 to 4 frames, etc., so that the terminal device can capture a corresponding number of images at different exposure compensation values, such as after shooting 4 frames of images at EV0, cutting off to EV+1, and then shooting 5 frames of images at EV+1.
[0131] Alternatively, as another implementation, the number of images captured at each control exposure compensation value and the number of images captured at the reference exposure compensation value may also be determined according to the sensitivities corresponding to different exposure compensation values.
[0132] Among them, the sensitivity can be calculated based on the exposure compensation value. The number of images to be taken corresponding to different sensitivities can be pre-set in the terminal device. In this way, when taking images at different exposure compensation values, the sensitivity corresponding to the exposure compensation value can be obtained first, and then the corresponding number of images to be taken can be obtained. Generally speaking, from EV+2 to EV-2, the corresponding sensitivity decreases successively. Therefore, the corresponding number of taken images will also be retrieved successively. For example, the number of images taken for a certain scene, from EV+2 to EV-2, may correspond to 6, 5, 4, 3, and 2 respectively.
[0133] In the above embodiment, the method for fusing images at each exposure compensation value can be: determining the image with the highest clarity in at least two frames of images as a reference image, and then fusing the other frame images in the at least two frames of images into the reference image to obtain a corresponding fused image.
[0134] The fusion method for at least two frames of images under each comparison exposure compensation value and the fusion method for at least two frames of images under the baseline exposure compensation value can both be the above-mentioned fusion method, that is, one frame of image is selected as a reference image, and then other frame images are fused to the reference image, and finally a fused image is obtained by fusion.
[0135] Among them, the corresponding gradient algorithm can be used to determine the image with the highest clarity, such as evaluating the image clarity based on the maximum value of the gradient edge, or evaluating the image clarity based on the point sharpness and the square gradient, or evaluating the image clarity based on the Brenner gradient function, etc. In this way, the clarity of each frame of the image can be evaluated, and then the image with the highest clarity is used as the reference image, so that after fusion, a fused image with good denoising can be obtained. The specific implementation process of the clarity evaluation can refer to the relevant implementation process in the prior art, which will not be described in detail here.
[0136] The image fusion method mentioned above can be to fuse the pixel value of each pixel in other frame images with the pixel value of the corresponding pixel in the reference image, such as taking the average value of the pixel values of two corresponding pixels as the pixel value of the pixel in the fused image. According to this method, multiple frame images can be fused into one image.
[0137] In an embodiment of the present application, in order to perform better fusion and denoising processing on the image, the fusion method can be: determine the pixel difference between each other frame image and each pixel in the reference image, determine the fusion weight of each corresponding pixel in each other frame image based on the pixel difference, and then fuse the other frame images with each corresponding pixel in the reference image based on the fusion weight of each corresponding pixel in each other frame image to obtain the corresponding fused image.
[0138] For example, if Figure 3 As shown, Figure 3 Here are three frames of images captured at EV+1. The pixel values of each pixel in each image are as follows: Figure 3 As shown in , the image labeled b is used as the reference image. It should be noted here that before pixel fusion, since there may be deviations between each frame of image, the other two frames of image can be first registered to the reference image. The specific registration implementation process can adopt the existing related registration method, which will not be described in detail here.
[0139] Assumptions Figure 3The three images shown in are three registered images, and their pixels are aligned, such as pixel 1 in image a, pixel 2 in image b, and pixel 3 in image c. When these three pixels are fused, the pixel difference between pixel 1 and pixel 2, and the pixel difference between pixel 3 and pixel 2 can be calculated first, and the fusion weight can be calculated as (255-pixel difference) / 255. Of course, in practical applications, the fusion weight can be calculated in other ways, such as pixel difference / 255, which can be flexibly set according to actual needs. The fusion weight can be understood as the closer the pixels at the same position are, the greater the fusion weight. Therefore, based on the above method, the fusion weights of pixel 1 and pixel 3 can be obtained. For example, the pixel value of pixel 1 is 30, the pixel value of pixel 2 is 20, and the pixel value of pixel 3 is 40. At this time, the fusion weight corresponding to pixel 1 is (255-10) / 255=0.96, and the fusion weight corresponding to pixel 3 is (255-20) / 255=0.92. When these three pixels are fused, the calculation formula for the fused pixel value is: (pixel value of pixel 1 * corresponding fusion weight + pixel value of pixel 2 * corresponding fusion weight (since image b is the reference image, its fusion weight is 1) + pixel value of pixel 3 * corresponding fusion weight) / sum of the fusion weights of each pixel. According to this calculation formula, the pixel value of the above three pixels after fusion is 29.72.
[0140] It should be noted that the above pixel value fusion calculation formula is only an example. In actual applications, it can be fused based on other methods, such as first adding the fusion weights of each pixel value and then multiplying it by the sum of each pixel value. The specific fusion method based on fusion weights can be flexibly set based on actual needs. Of course, in actual applications, other fusion methods can also be used, such as taking the pixel mean as the fused pixel value, etc. The above fusion method can effectively extract the effective pixel points in each frame of the image, achieve a better denoising effect, and thus obtain a well-denoised fused image.
[0141] According to the above fusion method, each corresponding pixel point can be fused, so that the pixel value of each pixel point after fusion can be obtained as the pixel value of each pixel point in the fused image, such as Figure 3 The image d in the middle is the fused image after fusion.
[0142] The above-mentioned examples can be used for the fusion method of images at each exposure compensation value, that is, the method for obtaining the reference image by fusion of images at the reference exposure compensation value can also be used in the same way, which will not be described in detail one by one here.
[0143] In the above implementation process, pixel fusion is performed based on the fusion weight, which can effectively extract the highlight pixels in the image and achieve a good fusion of the highlight pixels and the low-light pixels.
[0144] In the above-mentioned process of fusing at least two frames of images under each comparison exposure compensation value, a motion mask image between at least two frames of images collected at each comparison exposure compensation value can also be obtained, and its implementation process is: obtaining pixel differences of each corresponding pixel point between other frame images and the reference image in the at least two frames of images collected at each comparison exposure compensation value, using the pixel differences of each corresponding pixel point as the initial motion mask image between each other frame image and the reference image, obtaining a total of multiple initial motion mask images, and then superimposing the multiple initial motion mask images to obtain the motion mask image between at least two frames of images collected at each comparison exposure compensation value.
[0145] For example, Figure 3 Take the three frames of images shown in as an example, first obtain the pixel difference between each corresponding pixel point of image a and image b (reference image), and the pixel difference between each corresponding pixel point of image c and image b. For example, if the pixel difference between pixel point 1 and pixel point 2 is 10, and the pixel difference between pixel point 3 and pixel point 2 is 20, then the pixel value of the pixel point in the initial motion mask image between image a and image b is 10, and the pixel value of the pixel point in the initial motion mask image between image c and image b is 20. At this time, two initial motion mask images can be obtained. Figure 4 As shown, the two initial motion mask images are then superimposed. The superposition can be understood as the addition of the pixel values of the corresponding pixels. For example, the pixel value obtained by adding the pixel value 10 to the pixel value 20 is 30, which is used as the pixel value of the corresponding pixel in the motion mask image between the original three frames of images. In this way, each corresponding pixel can be processed in the same way as above, and the pixel value of each pixel in the motion mask image is finally obtained as follows: Figure 4 as shown in .
[0146] In the above implementation process, the corresponding motion mask image is determined by obtaining the pixel difference between at least two frames collected according to each contrast exposure compensation value, so that the motion area in the image can be effectively determined.
[0147] In the above embodiment, in order to finally fuse the images, it is also necessary to first align the fused image at each contrast exposure compensation value with the reference image, and the alignment method can be: adjusting the image brightness of the fused image at each contrast exposure compensation value to the image brightness of the reference image, and then determining the alignment reference data for aligning the fused image at each contrast exposure compensation value after the image brightness adjustment with the reference image, and then aligning the fused image at each contrast exposure compensation value based on the alignment reference data to obtain the aligned image corresponding to each contrast exposure compensation value.
[0148] Among them, since the brightness of the images collected under each exposure compensation value is inconsistent, the brightness can be adjusted to the image brightness of the reference image during registration. The specific method can be to process the pixel values of the pixels. Since the pixel value difference between adjacent exposure compensation values is generally about 2 times, the pixel values of each pixel in the fused image corresponding to EV+1 can be divided by 2, so that the overall brightness of the fused image corresponding to EV+1 can be close to the brightness of the reference image corresponding to EV0. Alternatively, the histogram matching algorithm can be used to process the fused image corresponding to EV+1. The method of processing using the histogram matching algorithm can refer to the relevant implementation process in the prior art, which will not be introduced in detail here.
[0149] It should be understood that image registration is actually to find a spatial transformation to map one image to another, so that the pixels corresponding to the same position in the two images correspond one to one, thereby achieving the purpose of information fusion. In other words, registration is to find a mapping relationship between the fused image and the reference image, and the registration reference data is the mapping relationship. Based on such a mapping relationship, the two images can be registered.
[0150] Among them, there can be many methods for image registration, such as region-based matching method, image feature-based matching method, model-based matching method, etc. For different image registration methods, the registration reference data obtained are different. Taking the image feature-based registration method as an example, this method extracts the unchanged features in the image, such as edge points, line features, surface features, matrix features, etc. These features can be used as registration reference data for the registration of two images. Then, the registration of the two images can be achieved based on the registration reference data. For the specific registration process, please refer to the existing implementation method and will not be described in detail here.
[0151] The above-mentioned registration process can be exemplified as follows: if N control exposure compensation values include EV+2, EV+1, EV-1, and EV-2, and the benchmark exposure compensation value is EV0, during the above-mentioned registration, based on the registration reference data 1 (based on the above description of the registration reference data, it can be known that the registration reference data 1 refers to the feature data between the fused image corresponding to EV+2 and the benchmark image corresponding to EV0), the fused image corresponding to EV+2 is registered with the benchmark image corresponding to EV0; based on the registration reference data 2, the fused image corresponding to EV+1 is registered with the benchmark image corresponding to EV0; based on the registration reference data 3, the fused image corresponding to EV-1 is registered with the benchmark image corresponding to EV0; based on the registration reference data 4, the fused image corresponding to EV-2 is registered with the benchmark image corresponding to EV0.
[0152] In some other implementations, since the difference between the images collected by each exposure compensation value is large, if the fused image under the larger or smaller comparison exposure compensation value is directly registered to the reference image, it may cause a large registration error and be difficult to register. Therefore, when registering the fused image under each comparison exposure compensation value with the base station image, each fused image may be registered in sequence, such as first registering the fused image under the larger comparison exposure compensation value to the fused image under the smaller comparison exposure compensation value, and then registering it to the reference image. The specific implementation process may be as follows:
[0153] According to the order of the size of the N comparison exposure compensation values, the reference image is used as the i-th image in the sorting, the fused images under the j comparison exposure compensation values are included before the reference image, and the fused images under the k comparison exposure compensation values are included after the reference image. Then, during the registration, the first fused image can be first registered to the second fused image to obtain the first intermediate registration data and the first registered image, and x is taken as 1 to j-2 in sequence, and the x-th registered image is registered to the x+2-th fused image based on the x-th intermediate registration data. When x is j-1, the j-1-th registered image is registered to the reference image based on the j-1-th intermediate registration data. When registering k fused images, first register the kth fused image to the k-1th fused image to obtain the kth intermediate registration data and the kth registered image, and then take x as k to 1 in sequence, and register the xth registered image to the x-2th fused image based on the xth intermediate registration data. When x is 1, register the x+1th registered image to the reference image based on the x+1th intermediate registration data.
[0154] For example, if N control exposure compensation values include EV+2, EV+1, EV-1, and EV-2, and the benchmark exposure compensation value is EV0, during the above-mentioned registration, the fused image corresponding to EV+2 is registered with the benchmark image corresponding to EV1 based on the registration reference data 1, that is, the fused image corresponding to EV+2 is registered to the benchmark image corresponding to EV1, and the registered image corresponding to EV+2 can be obtained; the fused image corresponding to EV+1 is registered to the benchmark image corresponding to EV0 based on the registration reference data 2; then the registered image corresponding to EV+2 is registered to the benchmark image corresponding to EV0 based on the registration reference data 2, and the fused image corresponding to EV-1 is registered to the benchmark image corresponding to EV0 based on the registration reference data 3; the fused image corresponding to EV-2 is registered to the benchmark image corresponding to EV1 based on the registration reference data 4 to obtain the registered image corresponding to EV-2, and then the registered image corresponding to EV-2 is registered to the benchmark image corresponding to EV0 based on the registration reference data 3.
[0155] In the above, each fused image is registered to the reference image in sequence according to the size of the exposure compensation value, so as to reduce the difference between the fused images corresponding to each exposure compensation value, so that the registration accuracy is higher.
[0156] In the above implementation process, by registering the fused image corresponding to each comparison exposure compensation value to the reference image, subsequent ghost detection and image fusion can be facilitated.
[0157] Since the motion mask image corresponding to each control exposure compensation value cannot be directly registered with the reference image, the registration reference data obtained in the registration process of the fused image and the reference image can be reused, and the motion mask image corresponding to each control exposure compensation value can be registered based on the registration reference data to obtain the registered motion mask image corresponding to each control exposure compensation value.
[0158] It can be understood that when the fused image corresponding to the exposure compensation value is registered with the reference image, the registration reference data obtained can be understood as a mapping relationship between the two images. Therefore, the motion mask image can also be registered to the reference image based on the same mapping relationship. In this way, the motion mask image can be aligned with each corresponding pixel point in the reference image, and the motion mask image can be aligned with each corresponding pixel point in the fused image. This makes it easier to find the ghost area in the registered image in combination with the registered motion mask image.
[0159] In the above implementation process, the motion mask image is registered based on the registration reference data, so that when detecting ghosts, the pixel values at corresponding positions in the registered motion mask image can be effectively combined for detection.
[0160] In the above embodiment, after the images are registered, the ghost area in the registered images is detected. After the ghost area in the registered images is detected, the images can be fused into a target image. As an implementation method, when the reference image and N registered images are fused, the images can be fused in pairs. The new fused image formed after the fusion is fused with the unfused image, and finally fused into a target image. The specific fusion process is as follows:
[0161] Two images are selected as images to be fused from N+1 images formed by the reference image and N registered images and an intermediate fused image formed by the fusion of two images in the N+1 images. Then, one image to be fused is fused with another image to be fused according to the highlight area and non-ghost area in the ghost area of one image to be fused. The target image is obtained after all images in the N+1 images are fused once.
[0162] For example, if N+1 images include image 1, image 2, image 3, image 4 and image 5, the image to be fused may refer to one of the five images, or may refer to an intermediate fused image obtained by fusion of two of the five images, or may refer to an image obtained by fusion of the two of the five images and then fused with the remaining unfused images of the five images. For example, if image 1 and image 2 are first selected for fusion, the image obtained after fusion of image 1 and image 2 is called an intermediate fused image, and image 1 and image 2 are used as images to be fused. If the intermediate fused image is further fused with image 3, the intermediate fused image and image 3 are also called images to be fused. If the image obtained after fusion of the intermediate fused image and image 3 is further fused for the next time, the image obtained after fusion of the intermediate fused image and image 3 is also called an intermediate fused image. When participating in the next fusion, the intermediate fused image is used as the image to be fused. That is, the above-mentioned image to be fused may refer to an image in N+1 images, or may refer to an intermediate fused image. The intermediate fused image may be understood to refer to an image obtained by fusing two images in N+1 images, or may refer to an image obtained by further fusing two images in N+1 images with other images to be fused.
[0163] Among them, the above-mentioned 5 images include a registered image and a reference image. In the above-mentioned step S140, the ghost area in each registered image is obtained. During fusion, if it is necessary to obtain the ghost area of the reference image, the ghost area in the reference image can be obtained according to the same ghost area detection method. The specific implementation process of detecting the ghost area in the image is described in detail in the subsequent embodiments and will not be described in detail here.
[0164] After determining the ghost area in each image, the highlight area can also be determined from the ghost area. The highlight area refers to some areas where the pixel value is greater than the threshold. The image brightness of these areas is higher. Since the exposure compensation value corresponding to each image is different, that is, the brightness is different, when fusion is performed, in order to obtain an image with a better fusion effect, the pixel points in the darker area of the image with a large exposure compensation value and the pixel points in the brighter area of the image with a small exposure compensation value are retained as much as possible. Therefore, when fusion is performed, the two images can be fused based on the highlight area and non-ghost area in the ghost area of one image.
[0165] Taking the fusion of image 1 and image 2 as an example, during the fusion, one image can be selected as the background image. If image 1 is used as the background image, the highlight area and non-ghost area in the ghost area in image 1 can be obtained, and other areas in image 2 except the area corresponding to the highlight area in image 1 can be obtained, and the fusion is performed based on the pixel points of these areas in the two images.
[0166] When one of the images to be fused is an intermediate fused image, its ghost area is the area corresponding to the ghost area of the background image in the two images involved in the fusion, and its highlight area is the area corresponding to the highlight area in the background image. For example, when image 1 and image 2 are fused, image 1 is selected as the background image, and image 2 is the non-background image. The background image can be understood as a reference image, that is, when image 1 and image 2 are fused, the fusion is based on the highlight area, non-ghost area of image 1 and other areas in image 2. When image 1 and image 2 are fused to obtain an intermediate fused image a, the ghost area in the intermediate fused image a is the area corresponding to the ghost area in image 1, and its highlight area is also the area corresponding to the highlight area in image 1. When the intermediate fused image a is further fused with image 3 to obtain an intermediate fused image b, if the intermediate fused image a is selected as the background image, the ghost area in the intermediate fused image b is the area corresponding to the ghost area in the intermediate fused image a, and its highlight area is also the area corresponding to the highlight area in the intermediate fused image a.
[0167] In the above implementation process, by fusing the highlight area in the ghost area in the image, the highlight area pixels in the images with different brightness can be better fused, so that the clarity of the fused image is higher.
[0168] As an implementation manner, according to the highlight area and the non-ghost area in the ghost area of one image to be fused, the method of fusion of the two images to be fused may be:
[0169] The pixel points of the highlight area in one image to be fused and the pixel points of the area corresponding to the non-highlight area in the ghost area of the one image to be fused in the other image to be fused are used as the pixel points of the corresponding area in the image obtained after the two images to be fused, and the pixel points of other areas except the ghost area in one image to be fused are fused with the pixel points of the area corresponding to other areas in the other image to be fused.
[0170] For example, when fusing two images to be fused (such as image 1 and image 2), if image 1 is selected as the background image and image 2 is the non-background image, then during fusion, the pixels in the highlight area of image 1 are retained, that is, the pixels in the area corresponding to the highlight area in the image obtained after fusion are the pixels in the highlight area of image 1, and the pixels in the area corresponding to the non-highlight area in the ghost area of image 1 in image 2 are retained, that is, the pixels in the area corresponding to the non-highlight area in the ghost area of image 1 in the image obtained after fusion are the pixels in the area corresponding to the non-highlight area in the ghost area of image 1 in image 2. For the pixels in other non-ghost areas, the pixels in the non-ghost area of image 1 are fused with the pixels in the area corresponding to the non-ghost area of image 2, so that the fused image can be obtained, which is called the intermediate fused image. The intermediate fused image can also continue to be fused with other unfused images, such as image 3, and then an updated intermediate fused image is obtained. The fusion process is similar to the process of image 1 and image 2, and will not be repeated here.
[0171] In this way, two images can be randomly selected for fusion, and then an intermediate fusion image can be obtained, and then the intermediate fusion image can be fused with other images, and finally fused into a target image. Of course, in order to obtain a better fusion effect, the two images randomly selected for fusion can be adjacent images, such as image 2 and image 3, or image 3 and image 4, and then the intermediate fusion image can be fused with its adjacent images, such as image 2 and image 3 can be fused with image 1 or image 4 after obtaining the intermediate fusion image.
[0172] In order to achieve the fusion of images across exposure compensation values and obtain an image with better fusion effect, the fusion standard can be to retain the pixels of the bright area in the image with a small exposure compensation value as much as possible, and to retain the pixels of the dark area in the image with a large exposure compensation value. Therefore, the exposure compensation value corresponding to the ghost area in the above-mentioned image to be fused is smaller than the exposure compensation value corresponding to the ghost area in the other image to be fused.
[0173] It can be understood that when the image to be fused is an image among N+1 images, its ghost area is the ghost area of the image itself. When the image to be fused is an intermediate fused image, such as image a obtained by fusing image 1 and image 2 mentioned above, the ghost area of image a is the area corresponding to the ghost area of image 1 (background image), and the exposure compensation value of image a is the exposure compensation value corresponding to image 1. For example, the exposure compensation value of image 1 is EV1, and the exposure compensation value corresponding to image a is also EV1.
[0174] In the above implementation process, by fusing in the above manner, the pixels in the highlight areas of images with different brightness can be well fused, so that the clarity of the fused image is higher.
[0175] In the above-mentioned process of fusing pixel points of other areas outside the ghost area in one image to be fused with pixel points of areas corresponding to other areas in another image to be fused can be: select one of the image to be fused and the other image to be fused as the background image, and the other image as the non-background image, and then determine the pixel size of each corresponding pixel point in other areas except the ghost area in the background image, and then determine the fusion weight of each corresponding pixel point in other areas in the background image and the fusion weight of each corresponding pixel point in the area corresponding to other areas in the non-background image based on the pixel size, and then based on the fusion weight of each corresponding pixel point in other areas in the background image and the fusion weight of each corresponding pixel point in the area corresponding to other areas in the non-background image, fuse each corresponding pixel point in other areas in the background image with each corresponding pixel point in the area corresponding to other areas in the non-background image.
[0176] For example, if one image to be fused is image 1, and the other image to be fused is image 2, when fusion is performed, one image can be first selected from the two images as the background image, wherein any one of the images can be selected as the background image, and of course the background image can also be selected according to certain rules, such as when all images are fused, each time an image with a small exposure compensation value is selected as the background image, or each time an image with a large exposure compensation value is selected as the background image. For example, after image 1 and image 2 are fused to obtain image a, image 1 is selected as the background image (the exposure compensation value of image 1 is less than the exposure compensation value of image 2). If image a is continued to be fused with image 3, image a is selected as the background image (because the exposure compensation value of image a is less than the exposure compensation value of image 3). Alternatively, when fusion is performed twice, image 2 and image 3 can also be selected as the background images. Therefore, in this way, one of the images can be selected as the background image, and of course the other image is used as the non-background image.
[0177] After the background image and the non-background image are determined, during fusion, fusion can be performed based on the fusion weight of each pixel point, and the fusion weight can be determined based on the pixel size of each pixel point in the background image. The method for determining the fusion weight based on the pixel size of each pixel in the background image can be: for example, the two corresponding pixels in image 1 (if image 1 is selected as the background image at this time) and image 2 are pixel 1 (in image 1) and pixel 2 (in image 2). At this time, since image 2 is brighter than image 1, for the pixel fusion of the non-ghost area, the pixels in the bright image are retained as much as possible, so the fusion weight of the corresponding area of image 2 should be larger. If the pixel value of pixel 1 is 20 and the pixel value corresponding to pixel 2 is 30, the calculation formula of the fusion weight can be: the fusion weight corresponding to pixel 1 = 20 / 255 = 0.08, the fusion weight corresponding to pixel 2 = (255-20) / 255 = 0.92, then the pixel value of the corresponding pixel in the image obtained after fusion is 20*0.08+30*0.92 = 29.2, and the processing of pixels in other non-ghost areas can also be done in the same way, and the pixel values of each pixel in the highlight area of image 1 can be directly used as the pixel values of the corresponding pixels in the obtained fused image. The process can be as follows Figure 5 As shown, Figure 5 In the obtained fused intermediate fused image, the pixel points of the area corresponding to the highlight area in image 1 are the pixel points of the highlight area in image 1, and the pixel points of the area corresponding to the non-highlight area in image 1 (the area other than the highlight area in the ghost area in image 1) are the pixel points of the corresponding area in image 2.
[0178] In the above fusion process, if image 2 is selected as the background image, the calculation formula of its fusion weight can be: the fusion weight corresponding to pixel 1 = 30 / 255 = 0.12, the fusion weight corresponding to pixel 2 = (255-30) / 255 = 0.88, then the pixel value of the corresponding pixel in the image after fusion is 20*0.12+30*0.88=28.8. That is, when different images are selected as background images, the calculation method of the corresponding fusion weight is different. Of course, the fusion weight can be determined by the standard that the fusion weight of the dark area in the image with a large exposure compensation value is large, and the fusion weight of the dark area in the image with a small exposure compensation value is small. Therefore, according to this standard, the fusion weight can also be customized. For example, for the pixel points in the area corresponding to the non-ghost area in the above-mentioned image 2, the fusion weight is set to 0.8, and for the pixel points in the non-ghost area in the image 1, the fusion weight is 0.1.
[0179] Of course, in the above fusion, the fusion weight may not be required. When the pixels are fused, the mean of the pixel values of the corresponding pixels may be taken as the pixel mean of the fused pixels, or other fusion methods may be used.
[0180] If the intermediate fusion image obtained by the above-mentioned image 1 and image 2 is fused with image 3 again, the fusion process is similar to the above. During the fusion, if the intermediate fusion image is used as the background image, the pixels in the highlight area of the intermediate fusion image are retained, and the method for fusion of the pixels in other areas of the intermediate fusion image with the corresponding pixels in other areas of image 3 is the same as the above, which is not described in detail here.
[0181] In the above implementation process, pixel fusion is performed based on the fusion weight, which can effectively extract the highlight pixels in the image and achieve a good fusion of the highlight pixels and the low-light pixels.
[0182] However, in order to obtain a better fusion effect, the fusion can be performed in a certain order. As an implementation method, the N+1 images can also be sorted from large to small or from small to large according to the corresponding exposure compensation values, wherein the image with order i in the sorting is the i-th image. For example, the registered images include four registered images corresponding to the exposure compensation values of EV-2, EV-1, EV+1, and EV+2, and the reference image is the image corresponding to EV0. The images are sorted in the order of exposure compensation values, such as EV-2, EV-1, EV0, EV+1, and EV+2, or EV+2, EV+1, EV0, EV-1, and EV-2. Then, the N+1 images formed in the sorting order are: image 1 (EV-2), image 2 (EV-1), image 3 (EV0), image 4 (EV+1), and image 5 (EV+2), or: image 5 (EV+2), image 4 (EV+1), image 3 (EV0), image 2 (EV-1), and image 1 (EV-2).
[0183] During fusion, the process of selecting two images as the images to be fused can be: fuse the first image with the second image to form an intermediate fused image, sequentially take i as 3 to N+1, select the intermediate fused image formed by the fusion of all images before the i-th image in the sorting and the i-th image as the images to be fused. The fusion order of this method can be understood as: first fuse image 1 with image 2 to obtain image a, then fuse image a with image 3 to obtain image b, then fuse image b with image 4 to obtain image c, and finally fuse image c with image 5 to obtain the target image, or vice versa, first fuse image 5 with image 4 to obtain image a, then fuse image a with image 3 to obtain image b, then fuse image b with image 2 to obtain image c, and then fuse image c with image 1 to obtain the target image.
[0184] The above-mentioned images a, b and c become intermediate fused images.
[0185] The following describes the above fusion process in detail by ordering them as image 1 (EV-2), image 2 (EV-1), image 3 (EV0), image 4 (EV+1), and image 5 (EV+2). These images are called images to be fused in the above fusion process, and the following descriptions are all described by the image numbers themselves.
[0186] For example, if image 1 and image 2 are selected for fusion first (at this time, image 1 and image 2 are referred to as images to be fused in the above embodiment), if image 1 is used as the background image in the above embodiment, the highlight area of the ghost area in image 1 is first determined during fusion. Then, during fusion, the pixel points of the highlight area in image 1 are used as the pixel points of the corresponding area in image a obtained after fusion, and the pixel points of the area corresponding to the non-highlight area of image 1 in image 2 are used as the pixel points of the corresponding area in image a obtained after fusion. When the pixel points of the non-ghost area in image 1 are fused with the pixel points of the non-ghost area in image 2 corresponding to the non-ghost area in image 1, they can be fused according to the fusion weights introduced in the above embodiment.
[0187] According to the above method, the intermediate fused image (i.e. the above image a, also referred to as the image to be fused in the above embodiment) is further fused with image 3, and then an intermediate fused image is obtained to be further fused with image 4, and finally a completely fused target image is obtained.
[0188] It can be understood that if the N+1 images to be fused are sorted from large to small according to the exposure compensation value, the fusion process is as follows: image 5 is fused with image 4 to obtain an intermediate fused image, and then the intermediate fused image is fused with image 3 to obtain an intermediate fused image, which is then fused with image 2 to obtain another intermediate fused image, which is then fused with image 1 to obtain the final fused target image. The specific fusion process can refer to the fusion process in the above example, and will not be repeated here.
[0189] In the above implementation process, the images are fused in sequence according to the size of the exposure compensation values, so that images with different brightness can be better fused.
[0190] As another implementation, during fusion, the process of selecting two images as images to be fused can also be: i is taken as an odd number from 1 to N+1 in sequence, and then the adjacent i-th image and i+1-th image in the N+1 images are fused in sequence to form an intermediate fused image, and then two adjacent intermediate fused images are selected in sequence as images to be fused in the order in which the intermediate fused images are obtained.
[0191] For example, the order of fusion can be understood as follows: first, image 1 and image 2 are fused to obtain image a, then image 3 and image 4 are fused to obtain image b, then image a and image b are fused to obtain image c, and finally image c and image 5 are fused to obtain the target image. Or vice versa, first, image 5 and image 4 are fused to obtain image a, then image 3 and image 2 are fused to obtain image b, then image a and image b are fused to obtain image c, and finally image c is fused to image 1 to obtain the target image.
[0192] The fusion process can refer to the corresponding process of the above embodiment, and will not be repeated here. It is worth noting that when image a and image b are fused, if image a is selected as the background image, the highlight area of image a can be the area corresponding to the highlight area of image 1 (image 1 is used as the background image when image 1 and image 2 are fused), and the non-highlight area is the area corresponding to the non-highlight area in image 1. The fusion of the two images can also be performed according to the fusion method described in the above embodiment.
[0193] In the above implementation process, adjacent images with different brightness are fused, so as to reduce the brightness error during fusion and achieve better fusion effect.
[0194] In addition, the above-mentioned fusion process is fused in the order of exposure compensation value from small to large or from large to small. In actual situations, it is also possible to fuse from both sides with the reference image as the center, or to fuse to both sides, such as fusing EV-2 with EV-1 to obtain an intermediate fusion image, and then fusing the intermediate fusion image with EV0 to obtain another intermediate fusion image, fusing EV+2 with EV+1 to obtain an intermediate fusion image, fusing the intermediate fusion image with EV0 to obtain another intermediate fusion image, and finally fusing the two intermediate fusion images to obtain the final target image; or, fusing EV0 with EV-1 to obtain an intermediate fusion image, and then fusing the intermediate fusion image with EV-2 to obtain another intermediate fusion image, fusing EV0 with EV+1 to obtain an intermediate fusion image, and then fusing the intermediate fusion image with EV+2 to obtain another intermediate fusion image, and finally fusing the two intermediate fusion images to obtain the final target image.
[0195] It should also be noted that, in the above-mentioned fusion process, when selecting the background image, you can select the image with a small exposure compensation value as the background image each time, or select the image with a large exposure compensation value as the background image. In this way, in the fusion process, you can select the background image according to a unified standard, so that images with different exposure compensation values can be better fused.
[0196] By fusing in the above manner, the pixels in the highlight areas of images with different brightness can be well fused, so that the clarity of the fused image is higher.
[0197] In the above embodiment, since the pixel values of the pixels in the ghost area in the corresponding image to be fused may be 0 or greater than 1, in order to eliminate noise interference and accurately detect the ghost area, the ghost area in the image can be determined in the following way: binarize the initial ghost area in the corresponding image to be fused to obtain a binary ghost area, divide the binary ghost area to obtain multiple non-adjacent ghost areas.
[0198] Among them, the binarization processing can be understood as processing the pixel values in the initial ghost area, such as changing the pixel values of the pixels in the initial ghost area that are greater than a preset threshold (such as set to 20) to 1, and changing the pixel values that are less than or equal to the preset threshold to 0. In this way, the initial ghost area in the image to be fused can be divided into multiple non-adjacent ghost areas based on the area with a pixel value of 1. The division method can adopt the existing related image segmentation algorithm, which will not be introduced in detail here.
[0199] In the above implementation process, binarization is performed on the ghost area, which can effectively filter out the noise in the ghost area.
[0200] As an implementation mode, when the image to be fused is a registered image, the method for obtaining the initial ghost area in the registered image can be: based on the pixel deviation between the registered image and the reference image at each control exposure compensation value, and the pixel value in the registered motion mask image corresponding to each control exposure compensation value, determine the initial ghost area in the registered image at each control exposure compensation value.
[0201] Among them, the pixel difference of each corresponding pixel point between the registered image and the reference image under each comparison exposure compensation value can be determined first, and then it is judged whether the pixel difference of each corresponding pixel point is greater than a first threshold, and whether the pixel value of the corresponding pixel point in the registered motion mask image corresponding to each comparison exposure compensation value is greater than a second threshold. If the pixel value difference of the pixel point is greater than the first threshold and the pixel value of the corresponding pixel point in the registered motion mask image is greater than the second threshold, then the position of the corresponding pixel point in the registered image under the corresponding comparison exposure compensation value is determined to be the initial ghost area.
[0202] For example, taking the control exposure compensation value as EV+1 and the benchmark exposure compensation value as EV0, since the pixels in the registered image corresponding to EV+1, the benchmark image, and the registered motion mask image corresponding to EV+1 have been aligned in the above-mentioned registration process, taking the three aligned pixels as an example, pixel 1 in the registered image corresponding to EV+1, pixel 2 in the benchmark image, and pixel 3 in the registered motion mask image corresponding to EV+1 are aligned pixels. If the pixel value of pixel 1 is 30, the pixel value of pixel 2 is 45, and the pixel value of pixel 3 is 22, in this case, the pixel difference between pixel 1 and pixel 2 is 15. If the first threshold is 10 and the second threshold is 12, then after comparison, it is found that the pixel difference between pixel 1 and pixel 2 is greater than the first threshold, and the pixel value of pixel 3 is greater than the second threshold. Therefore, it can be determined that the position of pixel 1 in the registered image corresponding to EV+1 is the initial ghost area, and the same is true for the detection of other pixels. In this way, it can be determined which pixels in the registered image are ghost points, that is, which areas are the initial ghost areas.
[0203] It should be noted that when the pixel difference is less than or equal to the first threshold, and / or the pixel value of the corresponding pixel in the registered motion mask image is less than or equal to the second threshold, it is determined that the position of the corresponding pixel in the registered image is not the initial ghost area.
[0204] The specific values of the first threshold and the second threshold can be flexibly set according to actual needs. In some cases, the values of the first threshold and the second threshold can be the same or different.
[0205] According to the above method, pixel comparison can be performed on each registered image with the reference image, and the initial ghost area in each registered image can be determined in combination with the corresponding registered motion mask image. The detection method of the initial ghost area in other registered images can refer to the above-mentioned detection method of the initial ghost area in the registered image corresponding to EV+1. For the sake of brevity, it will not be described in detail here.
[0206] It should also be noted that if fusion is required based on the ghost area of the reference image during the fusion process, the detection of the initial ghost area of the reference image can be: determine whether the pixel value of the corresponding pixel point in the reference image is greater than a first threshold, and determine whether the pixel value of the corresponding pixel point in the motion mask image corresponding to the reference image is greater than a second threshold. If the pixel value of the corresponding pixel point in the reference image is greater than the first threshold, and the pixel value of the corresponding pixel point in the motion mask image corresponding to the reference image is greater than the second threshold, then the position of the corresponding pixel point in the reference image is determined to be the initial ghost area.
[0207] In the above implementation process, ghost detection in the registered image is performed in combination with the pixel values of the pixel points in the registered motion mask image. This can avoid the problem in the prior art of performing ghost detection only through the pixel difference between the two original images, resulting in a large number of highlight areas being mistakenly detected as ghost areas, thereby effectively improving the accuracy of ghost detection.
[0208] After determining the initial ghost area in each image, the initial ghost area can be processed as above and divided into multiple ghost areas. When fusing images, the highlight area can be determined from the multiple ghost areas, and the fusion can be performed based on the highlight area, non-highlight area and non-ghost area.
[0209] Among them, the highlight area in the ghost area can be determined in the following way: determine the average pixel value of the pixel points in each ghost area in the corresponding image to be fused; if the average pixel value of the corresponding ghost area is greater than a preset value, determine that the corresponding ghost area in the corresponding image to be fused is a highlight area.
[0210] After being divided into multiple ghost areas, the pixel average value of the pixels originally in each ghost area in the image to be fused is obtained. If it is greater than the preset value, the corresponding ghost area is determined to be a highlight area. At this time, when the two images are fused as mentioned above, the pixels in the highlight area are retained, while the pixels in the non-highlight area are not retained, and the pixels in the non-ghost area are fused with the pixels in the corresponding area in the other image.
[0211] In the above implementation process, by dividing the ghost area, the highlight area in the ghost area can be found more accurately.
[0212] As an implementation method, based on the above embodiment, after obtaining the target image, in order to achieve personalized adjustment of the target image, the target image can also be adjusted by a limited contrast adaptive histogram equalization algorithm, that is, the target image is adjusted by a CLAHE algorithm, so that the brightness and contrast of the target image can be adjusted individually, and finally an image with good overall brightness, contrast, noise and clarity can be obtained. The specific implementation process of adjusting the target image can refer to the process of adjusting the image by the CLAHE algorithm in the prior art, which will not be described in detail here.
[0213] Please refer to Figure 6 , Figure 6 2 is a structural block diagram of an image processing device 200 provided in an embodiment of the present application. The device 200 may be a module, program segment or code on an electronic device. It should be understood that the device 200 is similar to the above-mentioned Figure 2 The method embodiment corresponds to and can be executed Figure 2The various steps involved in the method embodiment and the specific functions of the device 200 can be found in the above description. To avoid repetition, the detailed description is appropriately omitted here.
[0214] Optionally, the device 200 includes:
[0215] The image acquisition module 210 is used to obtain a reference image at a reference exposure compensation value and fused images at N comparison exposure compensation values, wherein the reference image is obtained by fusion of at least two frames of images collected at the reference exposure compensation value; and the fused image is obtained by fusion of at least two frames of images collected at the comparison exposure compensation value;
[0216] A motion mask image acquisition module 220 is used to determine a motion mask image between at least two frames of images collected at each comparison exposure compensation value, wherein the pixel value of a pixel point in the motion mask image is used to represent the motion amplitude of the corresponding pixel point in at least two frames of images collected at each comparison exposure compensation value;
[0217] An image registration module 230 is used to register the fused image under each contrast exposure compensation value and the motion mask image corresponding to each contrast exposure compensation value with the reference image, respectively, to obtain a registered image corresponding to each contrast exposure compensation value and a registered motion mask image;
[0218] A ghost region detection module 240, configured to determine a ghost region in the registered image at each contrast exposure compensation value based on the registered image at each contrast exposure compensation value, the reference image, and the registered motion mask image corresponding to each contrast exposure compensation value;
[0219] The image fusion module 250 is used to fuse the registration images corresponding to the N comparison exposure compensation values and the reference image based on the ghost area to obtain a target image.
[0220] Optionally, the image fusion module 250 is used to:
[0221] Selecting two images as to-be-fused images from N+1 images formed by the reference image and the N registered images and an intermediate fused image formed by fusing two images from the N+1 images;
[0222] According to the highlight area and the non-ghost area in the ghost area of one image to be fused, the one image to be fused is fused with the other image to be fused, and the target image is obtained after all the images in the N+1 images are fused once;
[0223] Among them, when the image to be fused is an intermediate fused image, its ghost area is the area corresponding to the ghost area of the background image in the two images participating in the fusion, and its highlight area is the area corresponding to the highlight area in the background image.
[0224] Optionally, the image fusion module 250 is used to:
[0225] Taking the pixel points of the highlight area in the one image to be fused and the pixel points of the area corresponding to the non-highlight area in the ghost area of the one image to be fused in the other image to be fused as the pixel points of the corresponding area in the image obtained after the two images to be fused;
[0226] Fusing pixel points of other regions except the ghost region in the one image to be fused with pixel points of a region corresponding to the other region in the other image to be fused;
[0227] The exposure compensation value corresponding to the ghost area in the one image to be fused is smaller than the exposure compensation value corresponding to the ghost area in the other image to be fused.
[0228] Optionally, the image fusion module 250 is used to:
[0229] Selecting one of the image to be fused and the other image to be fused as a background image, and the other image as a non-background image;
[0230] Determine the pixel size of each corresponding pixel point in other areas of the substrate image except the ghost area;
[0231] Determine, based on the pixel size, a fusion weight of each corresponding pixel point in the other area in the substrate image and a fusion weight of each corresponding pixel point in the area corresponding to the other area in the non-substrate image;
[0232] Based on the fusion weights of each corresponding pixel point in the other areas in the background image and the fusion weights of each corresponding pixel point in the area corresponding to the other areas in the non-background image, the other areas in the background image and the corresponding pixels in the area corresponding to the other areas in the non-background image are fused.
[0233] Optionally, the image whose order among the N+1 images is i in descending order or ascending order according to the corresponding exposure compensation values is the i-th image, and the image fusion module 250 is used to:
[0234] Fusing the first image with the second image to form an intermediate fused image;
[0235] i is sequentially selected from 3 to N+1, and an intermediate fused image formed by fusion of all images before the i-th image in the order and the i-th image are selected as images to be fused.
[0236] Optionally, the image whose order among the N+1 images is i in descending order or ascending order according to the corresponding exposure compensation values is the i-th image, and the image fusion module 250 is used to:
[0237] i is taken as an odd number from 1 to N+1, and the adjacent i-th image and i+1-th image in the N+1 images are fused in sequence according to the sorting order to form an intermediate fused image;
[0238] According to the order in which the intermediate fusion images are obtained, two adjacent intermediate fusion images are selected in sequence as the images to be fused.
[0239] Optionally, when the image to be fused is one of the N+1 images, the ghost area in the corresponding image to be fused is determined in the following manner:
[0240] Binarize the initial ghost area in the corresponding image to be fused to obtain a binary ghost area;
[0241] The binary ghost region is divided to obtain a plurality of non-adjacent ghost regions.
[0242] Optionally, the highlight area in the ghost area of the corresponding image to be fused is determined in the following manner:
[0243] Determine the average pixel value of each pixel in the ghost area of the corresponding image to be fused;
[0244] If the average pixel value of the corresponding ghost area is greater than a preset value, it is determined that the corresponding ghost area in the corresponding image to be fused is a highlight area.
[0245] Optionally, when the image to be fused is the registered image, the initial ghost area in the corresponding registered image is determined by:
[0246] Based on the pixel deviation between the registered image at each comparison exposure compensation value and the reference image, and the pixel value in the registered motion mask image corresponding to each comparison exposure compensation value, an initial ghost area in the registered image at each comparison exposure compensation value is determined.
[0247] Optionally, determining the initial ghost area in the registered image at each comparison exposure compensation value based on the pixel deviation between the registered image at each comparison exposure compensation value and the reference image, and the pixel value in the registered motion mask image corresponding to each comparison exposure compensation value, comprises:
[0248] Determine the pixel difference of each corresponding pixel point between the registration image and the reference image at each comparison exposure compensation value;
[0249] Determine whether the pixel value difference of the corresponding pixel point is greater than a first threshold, and whether the pixel value of the corresponding pixel point in the registered motion mask image corresponding to each comparison exposure compensation value is greater than a second threshold;
[0250] If the pixel value difference of the corresponding pixel point is greater than the first threshold and the pixel value of the corresponding pixel point in the registered motion mask image is greater than the second threshold, the position of the corresponding pixel point in the registered image under the corresponding comparison exposure compensation value is determined to be the initial ghost area.
[0251] Optionally, the image registration module 230 is used to adjust the image brightness of the fused image at each control exposure compensation value to the image brightness of the reference image; determine the registration reference data for registering the fused image at each control exposure compensation value after the image brightness is adjusted with the reference image; and register the fused image at each control exposure compensation value based on the registration reference data to obtain the registered image corresponding to each control exposure compensation value.
[0252] Optionally, the image registration module 230 is used to register the motion mask image corresponding to each comparison exposure compensation value based on the registration reference data to obtain a registered motion mask image corresponding to each comparison exposure compensation value.
[0253] Optionally, the device 200 further includes:
[0254] The contrast adjustment module is used to adjust the target image by using a contrast-limited adaptive histogram equalization algorithm.
[0255] Optionally, the image acquisition module 210 is used to acquire at least two frames of images captured at each of the N control exposure compensation values and at least two frames of images captured at the baseline exposure compensation value for the same scene; fuse the at least two frames of images captured at each control exposure compensation value to obtain a fused image at the N control exposure compensation values, and fuse the at least two frames of images captured at the baseline exposure compensation value to obtain a baseline image at the baseline exposure compensation value.
[0256] Optionally, at least two frames of images are fused in the following manner:
[0257] Determine the image with the highest definition among the at least two frames of images as a reference image;
[0258] The other frame images of the at least two frame images are fused with the reference image to obtain a corresponding fused image.
[0259] Optionally, the image acquisition module 210 is used to determine the pixel difference between each other frame image and each corresponding pixel point in the reference image; determine the fusion weight of each corresponding pixel point in each other frame image based on the pixel difference; and fuse the other frame images with each corresponding pixel point in the reference image based on the fusion weight of each corresponding pixel point in each other frame image to obtain a corresponding fused image.
[0260] Optionally, each of the at least two frames of images collected for comparison with the exposure compensation value includes a reference image, and the motion mask image acquisition module 220 is used to:
[0261] Obtaining pixel differences of corresponding pixel points between other frame images in at least two frame images collected by comparing the exposure compensation value and the reference image;
[0262] Using the pixel difference of each corresponding pixel point as the initial motion mask image between each other frame image and the reference image, and obtaining a total of multiple initial motion mask images;
[0263] The multiple initial motion mask images are superimposed to obtain a motion mask image between at least two frames of images collected according to each comparison exposure compensation value.
[0264] Optionally, the number of images captured at each control exposure compensation value and the number of images captured at the reference exposure compensation value are determined according to the sensitivities corresponding to different exposure compensation values.
[0265] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0266] The present application provides a readable storage medium, wherein when the computer program is executed by a processor, the computer program is executed as follows: Figure 2 The method process in the method embodiment shown is executed by the electronic device.
[0267] The present embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining a reference image under a reference exposure compensation value and a fused image under N comparison exposure compensation values, wherein the reference image is obtained by fusion of at least two frames of images captured under the reference exposure compensation value; the fused image is obtained by fusion of at least two frames of images captured under the comparison exposure compensation value; determining a motion mask image between at least two frames of images captured at each comparison exposure compensation value, the motion mask image The pixel values of the pixels in the mask image are used to characterize the motion amplitude of the corresponding pixels in at least two frames of images collected at each comparison exposure compensation value; the fused image at each comparison exposure compensation value and the motion mask image corresponding to each comparison exposure compensation value are respectively registered with the reference image to obtain a registered image and a registered motion mask image corresponding to each comparison exposure compensation value; based on the registered image at each comparison exposure compensation value, the reference image and the registered motion mask image corresponding to each comparison exposure compensation value, a ghost area in the registered image at each comparison exposure compensation value is determined; based on the ghost area, the registered images corresponding to the N comparison exposure compensation values and the reference image are fused to obtain a target image.
[0268] In summary, the embodiments of the present application provide an image processing method, device, electronic device and readable storage medium, which obtains a motion mask image between at least two frames of images captured for each control exposure compensation value, and then aligns the motion mask image. Thus, the ghost area in the registered image under each control exposure compensation value can be accurately detected in combination with the registered motion mask image, thereby improving the accuracy of ghost area detection. In this way, when image fusion is performed based on the ghost area, the ghost can be eliminated well to obtain an image with higher clarity.
[0269] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0270] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0271] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0272] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.
[0273] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. An image processing method, It is characterized in that The method comprises: Obtaining a reference image at a reference exposure compensation value and fused images at N comparison exposure compensation values, wherein the reference image is obtained by fusing at least two frames of images captured at the reference exposure compensation value; and the fused image is obtained by fusing at least two frames of images captured at the comparison exposure compensation value; Determine a motion mask image between at least two frames of images collected at each comparison exposure compensation value, wherein pixel values of pixels in the motion mask image are used to characterize the motion amplitude of corresponding pixels in at least two frames of images collected at each comparison exposure compensation value; The fused image under each contrast exposure compensation value and the motion mask image corresponding to each contrast exposure compensation value are respectively registered with the reference image to obtain a registered image corresponding to each contrast exposure compensation value and a registered motion mask image; Determine a ghost area in the registered image at each comparison exposure compensation value based on the registered image at each comparison exposure compensation value, the reference image, and the registered motion mask image corresponding to each comparison exposure compensation value; Based on the ghost area, the registration images corresponding to the N comparison exposure compensation values and the reference image are fused to obtain a target image.
2. The method according to claim 1, It is characterized in that The step of fusing the registration images corresponding to the N comparison exposure compensation values and the reference image based on the ghost area to obtain a target image includes: Selecting two images as to-be-fused images from N+1 images formed by the reference image and the N registered images and an intermediate fused image formed by fusing two images from the N+1 images; According to the highlight area and the non-ghost area in the ghost area of one image to be fused, the image to be fused is fused with another image to be fused, and the target image is obtained after all the images in the N+1 images are fused once; Among them, when the image to be fused is an intermediate fused image, its ghost area is the area corresponding to the ghost area of the background image in the two images participating in the fusion, and its highlight area is the area corresponding to the highlight area in the background image.
3. The method according to claim 2, It is characterized in that The step of fusing the image to be fused with another image to be fused according to the highlight area and the non-ghost area in the ghost area of the image to be fused comprises: Taking the pixel points of the highlight area in the one image to be fused and the pixel points of the area corresponding to the non-highlight area in the ghost area of the one image to be fused in the other image to be fused as the pixel points of the corresponding area in the image obtained after the two images to be fused; Fusing pixel points of other regions except the ghost region in the one image to be fused with pixel points of a region corresponding to the other region in the other image to be fused; The exposure compensation value corresponding to the ghost area in the one image to be fused is smaller than the exposure compensation value corresponding to the ghost area in the other image to be fused.
4. The method according to claim 3, It is characterized in that The fusing pixel points of other areas except the ghost area in the one image to be fused with pixel points of an area corresponding to the other area in the other image to be fused includes: Selecting one of the image to be fused and the other image to be fused as a background image, and the other image as a non-background image; Determine the pixel size of each corresponding pixel point in other areas of the substrate image except the ghost area; Determine, based on the pixel size, a fusion weight of each corresponding pixel point in the other area in the substrate image and a fusion weight of each corresponding pixel point in the area corresponding to the other area in the non-substrate image; Based on the fusion weights of each corresponding pixel point in the other areas in the background image and the fusion weights of each corresponding pixel point in the area corresponding to the other areas in the non-background image, the other areas in the background image and the corresponding pixels in the area corresponding to the other areas in the non-background image are fused.
5. The method according to claim 2, It is characterized in that The image in which the N+1 images are sorted in descending order or in ascending order according to the corresponding exposure compensation values is i as the i-th image, and the selecting two images from the N+1 images formed by the reference image and the N registered images and the intermediate fused image formed by fusing two images of the N+1 images as the images to be fused includes: Fusing the first image with the second image to form an intermediate fused image; i is sequentially selected from 3 to N+1, and an intermediate fused image formed by fusion of all images before the i-th image in the order and the i-th image are selected as images to be fused.
6. The method according to claim 2, It is characterized in that The image in which the N+1 images are sorted in descending order or in ascending order according to the corresponding exposure compensation values is i as the i-th image, and the selecting two images from the N+1 images formed by the reference image and the N registered images and the intermediate fused image formed by fusing two images of the N+1 images as the images to be fused includes: i is taken as an odd number from 1 to N+1, and the adjacent i-th image and i+1-th image in the N+1 images are fused in sequence according to the sorting order to form an intermediate fused image; According to the order in which the intermediate fusion images are obtained, two adjacent intermediate fusion images are selected in sequence as the images to be fused.
7. The method according to claim 2, It is characterized in that When the image to be fused is one of the N+1 images, the ghost area in the corresponding image to be fused is determined in the following manner: Binarize the initial ghost area in the corresponding image to be fused to obtain a binary ghost area; The binary ghost region is divided to obtain a plurality of non-adjacent ghost regions.
8. The method according to claim 7, It is characterized in that Determine the highlight area in the ghost area of the corresponding image to be fused by the following method: Determine the average pixel value of each pixel in the ghost area of the corresponding image to be fused; If the average pixel value of the corresponding ghost area is greater than a preset value, it is determined that the corresponding ghost area in the corresponding image to be fused is a highlight area.
9. The method according to claim 7, It is characterized in that When the image to be fused is the registered image, the initial ghost area in the corresponding registered image is determined by: Based on the pixel deviation between the registered image at each comparison exposure compensation value and the reference image, and the pixel value in the registered motion mask image corresponding to each comparison exposure compensation value, an initial ghost area in the registered image at each comparison exposure compensation value is determined.
10. The method according to claim 9, It is characterized in that The method of determining an initial ghost region in the registered image at each comparison exposure compensation value based on a pixel deviation between the registered image at each comparison exposure compensation value and the reference image and a pixel value in the registered motion mask image corresponding to each comparison exposure compensation value comprises: Determine the pixel difference of each corresponding pixel point between the registration image and the reference image at each comparison exposure compensation value; Determine whether the pixel value difference of the corresponding pixel point is greater than a first threshold, and whether the pixel value of the corresponding pixel point in the registered motion mask image corresponding to each comparison exposure compensation value is greater than a second threshold; If the pixel value difference of the corresponding pixel point is greater than the first threshold and the pixel value of the corresponding pixel point in the registered motion mask image is greater than the second threshold, the position of the corresponding pixel point in the registered image under the corresponding comparison exposure compensation value is determined to be the initial ghost area.
11. The method according to claim 1, It is characterized in that The step of registering the fused image at each contrast exposure compensation value and the motion mask image corresponding to each contrast exposure compensation value with the reference image to obtain the registered image corresponding to each contrast exposure compensation value and the registered motion mask image comprises: Adjusting the image brightness of the fused image at each control exposure compensation value to the image brightness of the reference image; Determine registration reference data for registering the fused image at each comparison exposure compensation value after the image brightness is adjusted with the reference image; The fused image at each comparison exposure compensation value is registered based on the registration reference data to obtain a registered image corresponding to each comparison exposure compensation value.
12. The method according to claim 11, It is characterized in that The step of registering the fused image at each contrast exposure compensation value and the motion mask image corresponding to each contrast exposure compensation value with the reference image to obtain the registered image corresponding to each contrast exposure compensation value and the registered motion mask image comprises: The motion mask image corresponding to each comparison exposure compensation value is registered based on the registration reference data to obtain a registered motion mask image corresponding to each comparison exposure compensation value.
13. The method according to any one of claims 1 to 12, It is characterized in that After obtaining the target image, the method further includes: The target image is adjusted by using a contrast-limited adaptive histogram equalization algorithm.
14. The method according to any one of claims 1 to 13, It is characterized in that The obtaining of a reference image at a reference exposure compensation value and a fused image at N comparison exposure compensation values includes: Acquire at least two frames of images captured at each of the N reference exposure compensation values and at least two frames of images captured at the reference exposure compensation value for the same scene; At least two frames of images captured at each control exposure compensation value are fused to obtain fused images at the N control exposure compensation values, and at least two frames of images captured at the reference exposure compensation value are fused to obtain a reference image at the reference exposure compensation value.
15. The method according to claim 14, It is characterized in that At least two frames of images are fused in the following way: Determine the image with the highest definition among the at least two frames of images as a reference image; The other frame images of the at least two frame images are fused with the reference image to obtain a corresponding fused image.
16. The method according to claim 15, It is characterized in that The step of fusing other frame images of the at least two frame images into the reference image to obtain a corresponding fused image includes: Determine the pixel difference between each corresponding pixel in each other frame image and the reference image; Determine the fusion weight of each corresponding pixel point in each other frame image based on the pixel difference; Based on the fusion weights of the corresponding pixels in each of the other frame images, the other frame images are fused with the corresponding pixels in the reference image to obtain a corresponding fused image.
17. The method according to claim 14, It is characterized in that Each of the at least two frames of images collected with reference to the exposure compensation value includes a reference image, and determining a motion mask image between each of the at least two frames of images collected with reference to the exposure compensation value includes: Obtaining pixel differences of corresponding pixel points between other frame images in at least two frame images collected by comparing the exposure compensation value and the reference image; Using the pixel difference of each corresponding pixel point as the initial motion mask image between each other frame image and the reference image, and obtaining a total of multiple initial motion mask images; The multiple initial motion mask images are superimposed to obtain a motion mask image between at least two frames of images collected according to each comparison exposure compensation value.
18. The method according to claim 1, It is characterized in that The number of images captured at each control exposure compensation value and the number of images captured at the reference exposure compensation value are determined according to the sensitivities corresponding to different exposure compensation values.
19. An image processing device, It is characterized in that The device comprises: An image acquisition module, used to obtain a reference image at a reference exposure compensation value and fused images at N comparison exposure compensation values, wherein the reference image is obtained by fusion of at least two frames of images collected at the reference exposure compensation value; and the fused image is obtained by fusion of at least two frames of images collected at the comparison exposure compensation value; A motion mask image acquisition module, used to determine a motion mask image between at least two frames of images collected at each comparison exposure compensation value, wherein the pixel values of the pixels in the motion mask image are used to represent the motion amplitude of the corresponding pixel in at least two frames of images collected at each comparison exposure compensation value; An image registration module, used to register the fused image under each contrast exposure compensation value and the motion mask image corresponding to each contrast exposure compensation value with the reference image, respectively, to obtain a registered image corresponding to each contrast exposure compensation value and a registered motion mask image; a ghost region detection module, configured to determine a ghost region in the registered image at each comparison exposure compensation value based on the registered image at each comparison exposure compensation value, the reference image, and the registered motion mask image corresponding to each comparison exposure compensation value; An image fusion module is used to fuse the registration images corresponding to the N control exposure compensation values and the reference image based on the ghost area to obtain a target image.
20. An electronic device, It is characterized in that The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 18 is executed.
21. A readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 18 is executed.
Citation Information
Patent Citations
Image de-noising method and device, electronic device and computer readable storage medium
CN109348088A
Multi-frame image fusion method and device
CN110189285A
Image capture method and device
WO2019183813A1
Exposure control method and device, and electronic apparatus
WO2020038069A1
