Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: receiving a first image, a second image and a third image; the exposure duration of the first image is longer than that of the second image, and the exposure duration of the second image is longer than that of the third image; performing brightness alignment on the first image, the second image and the third image to obtain a first aligned image, a second aligned image and a third aligned image; performing ghosting removal processing on the first alignment image and the second alignment image; highlight fusion is carried out based on the first alignment image and the second alignment image after ghosting removal processing, and an intermediate fusion image is obtained; and performing highlight fusion based on the intermediate fusion image and the third alignment image to obtain an HDR image. According to the method and the device, the HDR image with a wider brightness range and more details is obtained by fusing a plurality of images with different exposure durations, and the HDR image with less ghosts and good image quality can be generated through ghosting removal processing.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and particularly relates to an image processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] The dynamic range of an image refers to the luminance difference between the darkest and brightest pixels in the image. High dynamic range (HDR) can present a wider luminance range and more details in the image.

[0003] When a terminal device takes a photo of a night scene, due to the dark scene, a photo taken with a short exposure time will be too dark and lose too many details, and a photo taken with a long exposure time will be overexposed in some areas and lose details and produce blur and artifacts due to motion, which will all affect the user's visual experience.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The present disclosure provides an image processing method and device, electronic equipment and storage medium.

[0006] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, comprising:

[0007] receiving a first image, a second image and a third image; wherein an exposure time of the first image is greater than the second image, and an exposure time of the second image is greater than the third image;

[0008] aligning the luminance of the first image, the second image and the third image to obtain a first aligned image, a second aligned image and a third aligned image after luminance alignment;

[0009] performing ghosting removal processing on the first aligned image and the second aligned image;

[0010] performing high light fusion based on the first aligned image and the second aligned image after ghosting removal processing to obtain an intermediate fusion image;

[0011] performing high light fusion based on the intermediate fusion image and the third aligned image to obtain a high dynamic range (HDR) image.

[0012] In some embodiments of the present disclosure, performing ghosting removal processing on the first aligned image and the second aligned image comprises:

[0013] The first aligned image and the second aligned image are input into a pre-trained ghost removal model to obtain the first aligned image and the second aligned image after ghost removal processing.

[0014] In some embodiments of this disclosure, the ghosting removal model is a student model obtained by network distillation based on a pre-trained teacher model.

[0015] In some embodiments of this disclosure, the first aligned image and the second aligned image are input into a pre-trained ghosting removal model to obtain the first aligned image and the second aligned image after ghosting removal, including:

[0016] Determine the reference image in the first and second aligned images;

[0017] The first aligned image, the second aligned image, and the reference image determination information are input into the ghost removal model to obtain the reference image after ghost removal processing.

[0018] In some embodiments of this disclosure, determining a reference image in the first alignment image and the second alignment image includes:

[0019] Determine the proportion of ghosting in the first aligned image;

[0020] In response to the ghost image proportion being less than or equal to a preset proportion threshold, the first aligned image is determined to be the reference image;

[0021] In response to the ghost image ratio being greater than a preset ratio threshold, the second aligned image is determined as the reference image.

[0022] In some embodiments of this disclosure, specular fusion is performed based on the first aligned image and the second aligned image after ghosting removal to obtain an intermediate fused image, including:

[0023] Based on the overexposed areas of the first aligned image after ghosting removal, a first specular blending mask is obtained;

[0024] Based on the first specular blending mask, the first aligned image and the second aligned image after ghosting removal are specularly blended to obtain the intermediate blended image.

[0025] In some embodiments of this disclosure, specular fusion is performed based on the first aligned image and the second aligned image after ghosting removal to obtain an intermediate fused image, including:

[0026] Based on the overexposed areas of the reference image after ghosting removal, a first specular blending mask is obtained;

[0027] Based on the first specular fusion mask, the reference image and the non-reference image after ghosting removal are specularly fused to obtain the intermediate fused image.

[0028] In some embodiments of this disclosure, specular fusion is performed based on the intermediate fused image and the third aligned image to obtain an HDR image, including:

[0029] Based on the overexposed areas of the intermediate fused image, a second highlight fusion mask is obtained;

[0030] Based on the second specular fusion mask, the intermediate fused image and the third aligned image are specularly fused to obtain the HDR image.

[0031] In some embodiments of this disclosure, the intermediate fused image and the third aligned image are specularly fused according to the second specular fusion mask to obtain the HDR image, including:

[0032] Based on the second specular fusion mask, the brightness of the intermediate fused image and the third aligned image are compared to determine the areas of abnormal brightness.

[0033] In response to the area of ​​the abnormal brightness region exceeding the area threshold, brightness correction is performed on the third aligned image;

[0034] Based on the second specular fusion mask, the intermediate fused image and the brightness-corrected third aligned image are specularly fused to obtain the HDR image.

[0035] In some embodiments of this disclosure, the step of comparing the brightness of the intermediate fused image and the third aligned image based on the second specular fusion mask to determine areas of abnormal brightness includes:

[0036] Based on the second specular fusion mask, a specular fusion boundary mask is generated;

[0037] Based on the aforementioned specular fusion boundary mask, the brightness of the intermediate fused image and the third aligned image are compared, and the boundary brightness gradient is calculated.

[0038] In response to the boundary brightness gradient exceeding a gradient threshold, the brightness anomaly region is identified.

[0039] In some embodiments of this disclosure, brightness correction is performed on the third aligned image, including:

[0040] The brightness of each pixel in the intermediate fused image and the third aligned image is compared to obtain a correction weight mask.

[0041] The brightness of the third aligned image is corrected based on the second specular fusion mask and the correction weight mask.

[0042] In some embodiments of this disclosure, brightness correction is performed on the third aligned image based on the second specular fusion mask and the correction weight mask, including:

[0043] Based on the overexposed areas of the third aligned image, a highlight white mask is obtained;

[0044] The brightness of the third aligned image is corrected based on the second specular blending mask, the specular white mask, and the correction weight mask.

[0045] In some embodiments of this disclosure, the brightness of each pixel of the intermediate fused image and the third aligned image is compared to obtain a corrected weight mask, and the method further includes:

[0046] The correction weight mask is feathered.

[0047] In some embodiments of this disclosure, an image processing method is provided, which further includes:

[0048] Detect highlight anomalies in the intermediate fused image and / or the HDR image to identify highlight anomaly regions;

[0049] Image retouching is performed on the aforementioned areas of abnormal highlight.

[0050] In some embodiments of this disclosure, image retouching of the highlighted abnormal region includes:

[0051] In response to the fact that the highlight anomaly region does not exceed the highlight anomaly threshold, image retouching is performed on the highlight anomaly region;

[0052] In response to the highlight anomaly region exceeding the highlight anomaly threshold, the image corresponding to the intermediate fused image is retained for the highlight anomaly region.

[0053] In some embodiments of this disclosure, receiving a first image, a second image, and a third image includes:

[0054] Preprocess the first image, the second image, and the third image;

[0055] Receive the preprocessed first image, second image, and third image;

[0056] The preprocessing includes at least one of the following: noise reduction and motion alignment.

[0057] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0058] An image receiving unit is configured to receive a first image, a second image, and a third image; wherein the exposure duration of the first image is greater than that of the second image, and the exposure duration of the second image is greater than that of the third image;

[0059] A brightness alignment unit is used to perform brightness alignment on the first image, the second image, and the third image to obtain a brightness-aligned first aligned image, a second aligned image, and a third aligned image.

[0060] The ghosting removal unit is used to perform ghosting removal processing on the first aligned image and the second aligned image.

[0061] The first specular fusion unit is used to perform specular fusion based on the first aligned image and the second aligned image after ghosting removal to obtain an intermediate fused image.

[0062] The second specular fusion unit is used to perform specular fusion based on the intermediate fused image and the third aligned image to obtain a high dynamic range (HDR) image.

[0063] According to a third aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0064] processor;

[0065] Memory used to store processor-executable instructions;

[0066] The processor is configured to implement the image processing method described in the first aspect above.

[0067] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the image processing method described in the first aspect.

[0068] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0069] This disclosure involves receiving a first image, a second image, and a third image; performing brightness alignment on the first, second, and third images to obtain a brightness-aligned first aligned image, a second aligned image, and a third aligned image; performing deghosting processing on the first and second aligned images; performing specular fusion on the deghosted first and second aligned images to obtain an intermediate fused image; and performing specular fusion on the intermediate fused image and the third aligned image to obtain a high dynamic range (HDR) image. By fusing multiple images with different exposure times, an HDR image with a wider brightness range and more detail is obtained. Furthermore, the deghosting processing enables the generation of high-quality HDR images with less ghosting.

[0070] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0071] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0072] Figure 1 This is a flowchart of an image processing method shown according to some embodiments of the present disclosure. Figure 1 .

[0073] Figure 2 This is a flowchart illustrating one implementation of step S106 according to some embodiments of the present disclosure.

[0074] Figure 3 This is a flowchart illustrating one implementation of step S202 according to some embodiments of the present disclosure.

[0075] Figure 4 This is a flowchart illustrating one implementation of step S302 according to some embodiments of the present disclosure.

[0076] Figure 5 This is a flowchart illustrating one implementation of step S108 according to some embodiments of the present disclosure.

[0077] Figure 6 This is a flowchart illustrating one implementation of step S108 according to some embodiments of the present disclosure.

[0078] Figure 7 This is a flowchart illustrating one implementation of step S110 according to some embodiments of the present disclosure.

[0079] Figure 8 This is a flowchart illustrating one implementation of step S704 according to some embodiments of the present disclosure.

[0080] Figure 9 This is a flowchart illustrating one implementation of step S802 according to some embodiments of the present disclosure.

[0081] Figure 10 This is a flowchart illustrating one implementation of step S804 according to some embodiments of the present disclosure.

[0082] Figure 11This is a flowchart illustrating one implementation of step S1004 according to some embodiments of the present disclosure.

[0083] Figure 12 This is a flowchart of an image processing method shown according to some embodiments of the present disclosure. Figure 2 .

[0084] Figure 13 This is a flowchart illustrating one implementation of step S1218 according to some embodiments of the present disclosure.

[0085] Figure 14 This is a flowchart illustrating a specific example of a process according to some embodiments of the present disclosure.

[0086] Figure 15 This is a block diagram of an image processing apparatus according to some embodiments of the present disclosure.

[0087] Figure 16 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0088] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0089] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0090] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0091] The applicant discovered that when the terminal device takes pictures of night scenes, due to the limitations of the camera sensor, it can only capture low dynamic range images with limited brightness variations. As a result, low dynamic range images usually contain overexposed or underexposed areas, thus losing a lot of detail and failing to meet the visual experience of the human eye.

[0092] In order to obtain image details and a wider brightness range when shooting night scenes, terminal devices can use multiple exposure times to acquire a set of low dynamic range images and then fuse them into a high dynamic range image. However, when the fusion area of ​​the inter-frame images undergoes relative motion and flicker is caused by artificial light sources, the final image acquired by the terminal device often has problems such as ghosting, voids, and layering of overexposed areas, which affects the user's shooting experience.

[0093] Figure 1 This is a flowchart of an image processing method shown according to some embodiments of the present disclosure. Figure 1 ,like Figure 1 As shown, the image processing method can be applied to electronic devices, including but not limited to smartphones, wearable devices, and smart tablets with shooting functions, as well as servers such as local servers and cloud servers that can provide shooting functions. The server can be deployed in a computer or a computer cluster consisting of multiple computers.

[0094] Figure 1 The image processing method shown includes the following steps.

[0095] In step S102, the first image, the second image, and the third image are received.

[0096] It should be noted that the exposure time of the first image is longer than that of the second image, and the exposure time of the second image is longer than that of the third image. The first, second, and third images are taken by the same device at adjacent time intervals for the same shooting scene, but with different exposure times.

[0097] In the exemplary embodiments of this disclosure, the first image is a long-exposure image, the second image is a short-exposure image, and the third image is an extremely short-exposure image. However, those skilled in the art will understand that the exposure time of the first, second, and third images is not limited, and the specific exposure time value is related to the image brightness. Accordingly, the image brightness level of the first image is higher than that of the second image, and the image brightness level of the second image is higher than that of the third image. For example, the average image brightness of the second image may be one-quarter of the average image brightness of the first image, and the average image brightness of the third image may be one-sixtieth-fourth of the average image brightness of the first image. Those skilled in the art will understand that the above image brightness values ​​are merely examples and are not intended to limit the scope of protection of this disclosure.

[0098] In the exemplary embodiments of this disclosure, the first, second, and third images are RAW images. This leverages the information directly captured by the device's sensor in the RAW images to ensure high fidelity, flexibility, and lossless editing of the first, second, and third images, thereby guaranteeing the image quality of the HDR image output after image processing. It should be noted that RAW image format is an unprocessed data format captured and stored by a digital camera or image sensor. When the camera shutter is pressed, the sensor captures light and converts it into an electronic signal containing brightness and color information of the scene. In most cases, the camera's internal processor converts and compresses this raw data into the common JPEG format. This process includes applying white balance, color correction, sharpening, and compression. However, if shooting in RAW format is selected, the camera will not process this raw data in any way but will save it directly. The RAW image format retains as much original image information as possible, which is why it is called a "digital negative," similar to film in traditional photography.

[0099] Those skilled in the art will understand that the embodiments of this disclosure can also process first, second, and third images of other image formats to obtain HDR images, such as JPEG and JPG formats, which will not be described in detail here.

[0100] In some embodiments of this disclosure, step S102 specifically includes: preprocessing the first image, the second image, and the third image; and receiving the preprocessed first image, the second image, and the third image. It should be noted that the preprocessing includes at least one of the following: denoising processing and motion alignment processing. Denoising processing of the first image, the second image, and the third image aims to restore the original state of the noise-contaminated image or to a state as close to the original as possible. Noise in the image is removed using denoising methods such as spatial domain methods, frequency domain methods, wavelet transform, deep learning methods, or model-based methods to reduce or eliminate noise, thereby improving image quality. Motion alignment processing of the first image, the second image, and the third image addresses inconsistencies between frames in the image sequence caused by camera movement or object motion, correcting the images to make them visually aligned or matched.

[0101] This embodiment of the disclosure preprocesses the first image, the second image, and the third image to reduce interference factors in the received first image, the second image, and the third image, thereby reducing interference factors and processing difficulty in subsequent image processing and reducing the resource investment of electronic devices; it also ensures the image quality of the HDR image output by image processing, giving users a better image experience.

[0102] In step S104, the first image, the second image, and the third image are brightness aligned to obtain the brightness-aligned first aligned image, the second aligned image, and the third aligned image.

[0103] It should be noted that since the brightness of the first image is higher than that of the second image, and the brightness of the second image is higher than that of the third image, the brightness of the first, second, and third images needs to be aligned.

[0104] In an exemplary embodiment of this disclosure, the brightness of the second and third images can be adjusted using the first image as a reference. For example, the brightness gain can be calculated based on parameters such as exposure parameters and shutter speed when the first, second, and third images are input, and the second and third images can be adjusted based on the calculated brightness gain.

[0105] In step S106, ghosting removal is performed on the first aligned image and the second aligned image.

[0106] It should be noted that during the photography process, because capturing multiple frames of target images takes a certain amount of time, when a moving object appears within the shooting range, the object's image may appear in all N adjacent frames (N being an integer greater than 2), resulting in ghosting (or double images, phantom images, etc.) of the moving object in the stitched image. In this case, deghosting can be performed during the photography process.

[0107] It should be noted that due to the long exposure, the first aligned image may exhibit ghosting in some areas if there are moving objects in the scene. Furthermore, when the first and second aligned images are fused, the different positions of these moving objects in the two images can also cause blurred or overlapping images, i.e., ghosting. Therefore, it is necessary to perform ghosting removal processing on both the first and second aligned images before image fusion.

[0108] In step S108, specular fusion is performed on the first and second aligned images after ghosting removal to obtain an intermediate fused image.

[0109] In step S110, high-light fusion is performed based on the intermediate fused image and the third aligned image to obtain a high dynamic range (HDR) image.

[0110] It should be noted that the image information contained in images with different exposure times is different. Steps S108 and S110 perform highlight fusion on images with different exposure times to fill in the missing image details and information in the highlight parts of a single image, thereby improving the image details and obtaining a high-quality HDR image.

[0111] It's important to note that the first image has a longer exposure time than the second image, and the second image has a longer exposure time than the third image. This means the first image is sharper than the second, and the second image is sharper than the third. Therefore, firstly, the highlights of the first and second aligned images (after ghosting removal) are blended. This aims to use the sharpest image as the main subject, while the slightly less sharp second aligned image is used to supplement details during blending. Then, the highlights of the intermediate blended image and the third aligned image are blended, using the less sharp third aligned image to supplement less image detail. This sequential blending process not only minimizes the processing workload during blending but also ensures that the sharpest image is used as the main subject for image blending, guaranteeing the image quality of the final HDR image.

[0112] As can be seen from the above steps, the image processing method provided in this disclosure involves receiving a first image, a second image, and a third image; performing brightness alignment on the first, second, and third images to obtain a brightness-aligned first aligned image, a second aligned image, and a third aligned image; performing ghosting removal processing on the first and second aligned images; performing specular fusion based on the ghosting-removed first and second aligned images to obtain an intermediate fused image; and performing specular fusion based on the intermediate fused image and the third aligned image to obtain a high dynamic range (HDR) image. By fusing multiple images with different exposure times, an HDR image with a wider brightness range and more detail is obtained, and through ghosting removal processing, a high-quality HDR image with less ghosting can be generated.

[0113] like Figure 2 The diagram shown is a flowchart illustrating the implementation process of step S106 provided in some exemplary embodiments of this disclosure, including the following steps.

[0114] In step S202, the first aligned image and the second aligned image are input into the pre-trained ghost removal model to obtain the first aligned image and the second aligned image after ghost removal processing.

[0115] In the exemplary embodiments of this disclosure, the ghosting removal model is a student model obtained by network distillation based on a pre-trained teacher model. Specifically, the teacher model is an AI model based on the Transformer architecture, which is a complex model. Simultaneously, a lightweight model is constructed as the student model for distillation learning to ensure ghosting removal accuracy while reducing computational cost and parameter count. It should be noted that network distillation (KD) is a machine learning technique primarily used to compress deep learning models. The basic idea is to extract knowledge from a large, complex model (the teacher model) and transfer this knowledge to a smaller, more efficient model (the student model), allowing the student model to achieve near-teacher model performance with lower computational cost. Embodiments of this disclosure can achieve ghosting removal models through network distillation using methods such as hard-label distillation, soft-label distillation, intermediate representation distillation, feature distillation, joint distillation, random network distillation, and relational distillation.

[0116] In an exemplary embodiment of this disclosure, the input to the aforementioned ghosting removal model is a set of images exposed at different times, and the output is an ideal image after the set of images are fully aligned. In the ideal image, there is no relative motion, and no ghosting is present after fusion. A rich training set can be constructed to ensure the accuracy of the ghosting removal model. Specifically, the training set may include a first training subset and a second training subset. The first training subset may be open-source image data from the network, and the second training subset may be image data collected and processed for night scene scenarios, particularly night scenes of vehicles driving on highways and night scenes of traffic lights on highways. This makes the ghosting removal model trained based on this training set more adaptable to night scene image processing and improves the model's robustness.

[0117] In the exemplary embodiments of this disclosure, since the images input to the ghost removal model have all undergone preprocessing such as motion alignment, that is, after coarse global alignment, when removing ghosts, it is only necessary to focus on the local small displacements at the highlights and the motion of special scenes such as rotation and jitter. In other words, the fitting difficulty of the constructed ghost removal model is greatly reduced. Therefore, it is only necessary to build a shallow feature extraction module to extract the relationship between local features of the image and align the local relationships between the images.

[0118] In the exemplary embodiments of this disclosure, the ghosting removal model is a lightweight model that consumes few resources and runs quickly. Compared to existing AI ghosting removal methods that consume a great deal of resources and have high power consumption, making them difficult to deploy on terminals, and where a large number of floating-point dot product calculations and complex network structures are very time-consuming, this ghosting removal model can be deployed on terminal devices to run in real time, thereby ensuring that users can quickly obtain processed images when using the terminal device.

[0119] like Figure 3 The diagram shown is a flowchart of the implementation process of step S202 provided in some exemplary embodiments of this disclosure, including the following steps.

[0120] In step S302, a reference image is determined in the first alignment image and the second alignment image.

[0121] In step S304, the first aligned image, the second aligned image, and the reference image determination information are input into the ghost removal model to obtain the ghost removal-processed reference image.

[0122] It should be noted that the reference image determination information indicates which of the first aligned image and the second aligned image is the reference image. Taking the reference image determination information as the first aligned image as an example, the process of inputting the first aligned image, the second aligned image, and the reference image determination information into the ghosting removal model to obtain the ghosting removal reference image includes: using the first aligned image as a reference, performing motion alignment on the second aligned image to eliminate the ghosting caused by the fusion between the first and second aligned images, thus obtaining the ghosting removal reference image.

[0123] like Figure 4 The diagram shown is a flowchart illustrating the implementation process of step S302 provided in some exemplary embodiments of this disclosure, including the following steps.

[0124] In step S402, the proportion of ghosting in the first aligned image is determined.

[0125] In an exemplary embodiment of this disclosure, the proportion of the ghost region in the first aligned image to the total image area is determined. This can be determined by determining the proportion of the ghost region in the first aligned image caused by the motion of a moving object due to long exposure, or by determining the proportion of the ghost region appearing when the first and second aligned images are fused.

[0126] In step S404, in response to the ghost image proportion being less than or equal to a preset proportion threshold, the first aligned image is determined as the reference image.

[0127] In step S406, in response to the ghost image proportion being greater than a preset proportion threshold, the second aligned image is determined as the reference image.

[0128] In the exemplary embodiments of this disclosure, due to the long exposure characteristics of the first aligned image, the image contains richer information. Typically, the first aligned image is selected as the reference image, and the second aligned image is used as a supplement. However, when the ghosting ratio is greater than a preset ratio threshold, it indicates that the first aligned image is not suitable as the reference image, meaning that if the first aligned image is used as the reference image, there are many areas that need to be modified or filled. In this case, the second aligned image can be used as the reference image to reduce the processing workload of the ghosting removal model.

[0129] For example, when the ghosting percentage is 18% and the preset percentage threshold is 20%, the first aligned image is determined as the reference image; and when the ghosting percentage is 23% and the preset percentage threshold is 20%, the first aligned image is determined as the reference image. Those skilled in the art will understand that the aforementioned preset percentage threshold values ​​are merely examples and are not intended to limit the scope of protection of this disclosure. These values ​​can be set according to actual needs, such as based on feedback from the ghosting removal model's processing speed.

[0130] In an exemplary embodiment of this disclosure, in response to a ghosting percentage exceeding a first preset percentage threshold, the second aligned image is deghosted to obtain an intermediate fused image. The first preset percentage threshold is much larger than the preset percentage threshold, for example, it could be 85%. When the ghosting percentage exceeds 85%, it indicates that the first aligned image is almost unusable, and there is no need to waste processing resources on image fusion. The second aligned image, after deghosting, can be used as the intermediate fused image for subsequent processing.

[0131] like Figure 5 The diagram shown is a flowchart illustrating the implementation process of step S108 provided in some exemplary embodiments of this disclosure, including the following steps.

[0132] In step S502, a first specular blending mask is obtained based on the overexposed areas of the first aligned image after ghosting removal.

[0133] In step S504, the first aligned image and the second aligned image after ghosting removal are specularly fused according to the first specular fusion mask to obtain an intermediate fused image.

[0134] In an exemplary embodiment of this disclosure, a first specular blending mask can be obtained by statistically analyzing the overexposed areas (i.e., the areas where a certain channel is truncated) of the first aligned image after ghosting removal using a brightness threshold. For the first specular blending mask, a portion of the second aligned image in the same area is replaced with a portion of the first aligned image after ghosting removal to obtain an intermediate blended image. Specifically, Laplacian pyramid fusion can be used to fuse the second aligned image and the first aligned image after ghosting removal in the Raw domain.

[0135] like Figure 6The diagram shown illustrates an implementation process of step S108 provided for other exemplary embodiments of this disclosure, including the following steps.

[0136] In step S602, a first specular blending mask is obtained based on the overexposed areas of the reference image after ghosting removal.

[0137] In step S604, the reference image and the non-reference image after ghosting removal are specularly fused according to the first specular fusion mask to obtain an intermediate fused image.

[0138] It should be noted that in some embodiments of this disclosure where the reference image is determined, the ghosting model outputs a ghosted reference image. A first specular blending mask can be obtained by statistically analyzing the overexposed areas of the ghosted reference image using a brightness threshold. For the first specular blending mask, portions of the non-reference image in the same area are replaced with portions of the ghosted reference image to obtain an intermediate blended image. In specific implementations, Laplacian pyramid fusion can be used to fuse the non-reference image and the ghosted reference image in the RAW domain.

[0139] In this embodiment, by determining a reference image in the first and second aligned images, ghosting caused by motion in the overexposed fusion region is avoided and reduced. This reduces the difficulty and processing range of the ghosting removal model, ensuring the effectiveness of ghosting removal. It also reduces the operational difficulty of the terminal device running the ghosting removal model, improving image processing speed and efficiency. Furthermore, the first aligned image is preferentially selected as the reference image, utilizing the high resolution of the long-exposure image to ensure that the fused image contains more information, thereby guaranteeing the quality of the final image.

[0140] like Figure 7 The diagram shown is a flowchart illustrating the implementation process of step S110 provided in some exemplary embodiments of this disclosure, including the following steps.

[0141] In step S702, a second highlight fusion mask is obtained based on the overexposed areas of the intermediate fused image.

[0142] In step S704, the intermediate fused image and the third aligned image are subjected to specular fusion according to the second specular fusion mask to obtain an HDR image.

[0143] In an exemplary embodiment of this disclosure, even after one specular fusion, areas of lost detail may still exist in the intermediate fused image. A first specular fusion mask can be obtained by statistically analyzing the overexposed areas of the intermediate fused image using a brightness threshold. For the second specular fusion mask, portions of the intermediate fused image in the same area are replaced with portions of the third aligned image to obtain an HDR image with rich detail. Specifically, Laplacian pyramid fusion can be used to fuse the intermediate fused image and the third aligned image in the RAW domain.

[0144] It should be noted that when shooting night scenes, the flickering of artificial light sources or sensor interference can cause differences in brightness relationships between different images. This can lead to abnormal highlight blending, resulting in highlight-dark layering or banding (a problem of inaccurate color representation in computer graphics). This issue is particularly pronounced when performing highlight blending between the intermediate blended image and the third-aligned image. In such cases, brightness correction of the third-aligned image is necessary before performing highlight blending between the intermediate blended image and the brightness-corrected third-aligned image.

[0145] In some embodiments of this disclosure, the severity of the brightness stratification or banding problem, and whether brightness correction is necessary, can be determined by whether the area of ​​the abnormal brightness region exceeds a threshold range. Accordingly, as... Figure 8 The diagram shown is a flowchart of the implementation process of step S704 provided in some exemplary embodiments of this disclosure, including the following steps.

[0146] In step S802, the brightness of the intermediate fused image and the third aligned image are compared according to the second specular fusion mask to determine the brightness abnormal area.

[0147] In step S804, in response to the area of ​​the abnormal brightness region exceeding the area threshold, brightness correction is performed on the third aligned image.

[0148] In step S806, the intermediate fused image and the brightness-corrected third aligned image are subjected to specular fusion based on the second specular fusion mask to obtain an HDR image.

[0149] It should be noted that by identifying areas of brightness abnormality and determining whether these areas exceed an area threshold, the severity of the brightness-dark layering or banding after fusion is assessed, and whether it meets the criteria for processing. If the area of ​​the brightness abnormality exceeds the area threshold, meaning that large areas of brightness-dark layering or banding occur, brightness correction of the third aligned image is required. Therefore, based on the second specular fusion mask, the intermediate fused image and the brightness-corrected third aligned image are specularly fused to obtain the HDR image.

[0150] In some embodiments of this disclosure, the specific process for determining the brightness abnormality region can be as follows: Figure 9 The diagram shown is a flowchart of the implementation process of step S802 provided in some exemplary embodiments of this disclosure, including the following steps.

[0151] In step S902, a specular fusion boundary mask is generated based on the second specular fusion mask.

[0152] In an exemplary embodiment of this disclosure, the second specular fusion mask can be expanded and etched, and the specular fusion anomalous region, i.e., the specular fusion boundary mask, can be obtained by subtracting the etched second specular fusion mask from the expanded second specular fusion mask.

[0153] In step S904, based on the specular fusion boundary mask, the brightness of the intermediate fused image and the third aligned image are compared, and the boundary brightness gradient is calculated.

[0154] In an exemplary embodiment of this disclosure, a 3×3 edge gradient operator, such as the Sobel operator, can be designed, and the brightness relationship can be used to compare the brightness of the intermediate fused image and the third aligned image to calculate the boundary brightness gradient.

[0155] In step S906, in response to the boundary brightness gradient exceeding the gradient threshold, a brightness anomaly region is identified.

[0156] In an exemplary embodiment of this disclosure, the gradient threshold described above represents a threshold for excessively large boundary brightness gradients. When the gradient threshold is exceeded, it proves that the pixel brightness variation in this region is large, and this region can be identified as a brightness abnormal region.

[0157] It should be noted that during the brightness correction process, a correction weight mask needs to be calculated to correct the brightness value of each pixel individually. For example... Figure 10 The diagram shown is a flowchart illustrating the implementation process of step S804 provided in some exemplary embodiments of this disclosure, including the following steps.

[0158] In step S1002, the brightness of each pixel in the intermediate fused image and the third aligned image is compared to obtain the corrected weight mask.

[0159] In an exemplary embodiment of this disclosure, the brightness of the entire image, including the intermediate fused image and the third aligned image, is compared pixel by pixel to determine the brightness ratio of each pixel in order to obtain a correction weight mask.

[0160] In some exemplary embodiments of this disclosure, step S1002 further includes feathering the correction weight mask to soften the edges of the selected area, making the color transition more natural and smooth, and preventing the image edges from appearing harsh or abrupt, thereby enhancing the visual effect and professional feel of the fused image.

[0161] In step S1004, the brightness of the third aligned image is corrected based on the second specular fusion mask and the correction weight mask.

[0162] In some embodiments of this disclosure, there may be highlighted white areas in the third aligned image. If these areas are also brightness corrected, it may cause fusion anomalies. Therefore, brightness correction may not be performed on these areas. Figure 11 The diagram shown is a flowchart of the implementation process of step S1004 provided in some exemplary embodiments of this disclosure, including the following steps.

[0163] In step S1102, a highlight white mask is obtained based on the overexposed area of ​​the third aligned image.

[0164] In an exemplary embodiment of this disclosure, a third-aligned image in RAW format can be converted into a true-color image in the RGB domain through an image conversion operation, and a white mask matrix in the image can be calculated based on the overexposed areas of the third-aligned image to obtain a highlight white mask.

[0165] In step S1104, the brightness of the third aligned image is corrected based on the second specular fusion mask, the specular white mask, and the correction weight mask.

[0166] In an exemplary embodiment of this disclosure, when performing brightness correction using the second highlight fusion mask and the highlight white mask, a parallel operation is performed, meaning that the highlight white portion corresponding to the highlight white mask does not need to be corrected, in order to prevent fusion anomalies from occurring.

[0167] The embodiments of this disclosure, through the above-described brightness correction processing, eliminate the inconsistency in brightness caused by changes in the energy of artificial light sources, thereby improving the image quality of the final image.

[0168] In some embodiments of this disclosure, the provided image processing method further includes: performing highlight anomaly detection on the intermediate fused image and / or HDR image to determine the highlight anomaly region; and performing image retouching on the highlight anomaly region.

[0169] Accordingly, such as Figure 12 The diagram illustrates a flow chart of an image processing method according to some embodiments of this disclosure. Figure 2 . Figure 12 In the image processing method shown, steps S1202 to S1208 and step S1214 are... Figure 1The steps S102 to S108 and step S110 in the image processing method shown correspond to each other and will not be repeated here.

[0170] In this embodiment of the disclosure, in Figure 1 Based on the image processing method shown, Figure 12 The image processing method shown may also include the following steps.

[0171] In step S1210, highlight anomaly detection is performed on the intermediate fused image to determine the highlight anomaly region.

[0172] In step S1212, image retouching is performed on the areas of abnormal highlights.

[0173] In step S1216, highlight anomaly detection is performed on the HDR image to determine the highlight anomaly region.

[0174] In step S1218, image retouching is performed on the areas of abnormal highlights.

[0175] It should be noted that the detection of highlight anomalies can be performed by checking whether there are small connected highlight regions in the image that have moved to the dark regions of adjacent frames. If so, holes will be created after fusion.

[0176] In an exemplary embodiment of this disclosure, the specific implementation process of step S1218 is as follows: Figure 13 As shown, the steps include the following.

[0177] In step S1302, in response to the fact that the highlight abnormality region does not exceed the highlight abnormality threshold, the highlight abnormality region is image retouched.

[0178] In step S1304, in response to the highlight anomaly region exceeding the highlight anomaly threshold, the image corresponding to the intermediate fused image of the highlight anomaly region is retained.

[0179] It should be noted that when the highlight anomaly area does not exceed the highlight anomaly threshold, the highlight movement is small in area or range, and can be filled and repaired by image inpainting. However, when the highlight anomaly area exceeds the highlight anomaly threshold, it indicates a large area or range of highlight movement. In this case, the inpainting algorithm cannot fill and repair the hole area, and dynamic range truncation is required. That is, for the highlight anomaly area, the image corresponding to the intermediate fused image is retained, while the image information of the corresponding area in the third aligned image is discarded to ensure the normal output of the HDR image.

[0180] This embodiment of the disclosure performs highlight anomaly detection on the image after two fusions, and performs image retouching after the highlight anomaly is detected, so as to eliminate the influence of the truncated outlier in the third aligned image on the HDR image and ensure the normal output of the HDR image.

[0181] In some embodiments of this disclosure, a set of denoised first images, second images, third images, and HDR images obtained after image processing can be recorded as a set of training data in an image processing training set. The image processing training set contains multiple sets of training data. Based on this image processing training set, a first deep learning model is constructed and trained to output an HDR image after fusing multiple RAW images, ensuring that the output HDR image is more accurate.

[0182] In some embodiments of this disclosure, a second deep learning model can be constructed and trained using deep learning algorithms to achieve dual processing of denoising and ghosting removal. For example, a high dynamic range RAW image with ghosting and noise removed can be generated by a neural network, thereby improving the processing speed of the terminal device and the image output speed of the terminal device.

[0183] To better illustrate the image processing method provided in the embodiments of this disclosure, a specific example is provided for further explanation, such as... Figure 14 The diagram shown is a flowchart illustrating the process of this specific example.

[0184] Figure 14 It includes the following steps.

[0185] S1401, based on the three frames of images after motion alignment and denoising, brightness alignment is performed to obtain long exposure image A, short exposure image B, and very short exposure image C.

[0186] S1402, Select image A as the reference image, and use the AI-deghost model to remove ghosting from images A and B to obtain the ghost-free image A'.

[0187] S1403, perform specular blending on image A' and image B to obtain intermediate image D.

[0188] S1404, perform highlight anomaly detection on image D, and perform image retouching on the highlight anomaly region to obtain image D'.

[0189] S1405, calculate the specular blending region between image D' and image C.

[0190] S1406, detect whether there is an excessive difference in image brightness between image D' and image C.

[0191] S1407, if so, perform brightness correction on image C to obtain the corrected image C'.

[0192] S1408, perform specular blending on images C' and D' to obtain image E0.

[0193] S1409, perform highlight anomaly detection on image E0, and perform image retouching on the highlight anomaly area to obtain image E0'.

[0194] S1410, Dynamic range truncation is performed on image E0'.

[0195] S1411 outputs HDR images.

[0196] S1412, if not, perform specular fusion on image C and image D' to obtain image E1.

[0197] S1413, perform highlight anomaly detection on image E1, and perform image retouching on the highlight anomaly area to obtain image E1'.

[0198] S1414, dynamically truncate the image E1'.

[0199] As can be seen from the above process, the algorithm in this specific example takes three images with different exposure times (very short exposure, short exposure, and long exposure form a set of inter-frame data) as input. Global alignment and denoising are performed on the three RAW images respectively, and a reference image is selected to reduce flickering bands in moving areas. The advantages of long and short exposure RAW images are combined for fusion, and alignment and deghosting operations are performed simultaneously to reduce the impact of camera shake during shooting, ensuring the quality of the final image. This improves the imaging effect of night scene shooting, outputting HDR images with less ghosting, no fusion anomalies, and a more stable dynamic range.

[0200] It should be noted that the acquisition, storage, use, and processing of data in this disclosed technical solution all comply with the relevant provisions of national laws and regulations.

[0201] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0202] Figure 15 This is a block diagram of an image processing apparatus according to some embodiments of the present disclosure. (Refer to...) Figure 15 The device includes: an image receiving unit 1501, a brightness alignment unit 1502, a ghost removal unit 1503, a first highlight fusion unit 1504, and a second highlight fusion unit 1505.

[0203] The image receiving unit 1501 is used to receive a first image, a second image, and a third image; wherein the exposure time of the first image is greater than that of the second image, and the exposure time of the second image is greater than that of the third image;

[0204] The brightness alignment unit 1502 is used to perform brightness alignment on the first image, the second image, and the third image to obtain the brightness-aligned first aligned image, the second aligned image, and the third aligned image.

[0205] Ghost removal unit 1503 is used to perform ghost removal processing on the first aligned image and the second aligned image;

[0206] The first specular fusion unit 1504 is used to perform specular fusion based on the first aligned image and the second aligned image after ghosting removal to obtain an intermediate fused image;

[0207] The second specular fusion unit 1505 is used to perform specular fusion based on the intermediate fused image and the third aligned image to obtain a high dynamic range (HDR) image.

[0208] In some exemplary embodiments of this disclosure, the image receiving unit 1501 is configured to: preprocess a first image, a second image, and a third image; and receive the preprocessed first image, second image, and third image; wherein the preprocessing includes at least one of the following: denoising processing and motion alignment processing.

[0209] In some exemplary embodiments of this disclosure, the ghosting removal unit 1503 is configured to input a first aligned image and a second aligned image into a pre-trained ghosting removal model to obtain a first aligned image and a second aligned image after ghosting removal.

[0210] It should be noted that the ghosting removal model is a student model obtained by using network distillation based on a pre-trained teacher model.

[0211] In some exemplary embodiments of this disclosure, the ghosting removal unit 1503 is configured to: determine a reference image in a first aligned image and a second aligned image; input the first aligned image, the second aligned image, and the reference image determination information into a ghosting removal model to obtain a ghosting removal-processed reference image.

[0212] In some exemplary embodiments of this disclosure, the ghost removal unit 1503 is configured to: determine the ghost ratio in the first aligned image; determine the first aligned image as a reference image in response to the ghost ratio being less than or equal to a preset ratio threshold; and determine the second aligned image as a reference image in response to the ghost ratio being greater than the preset ratio threshold.

[0213] In some exemplary embodiments of this disclosure, the first specular blending unit 1504 is configured to: obtain a first specular blending mask based on the overexposed areas of the first aligned image after ghosting removal; and perform specular blending on the first aligned image and the second aligned image after ghosting removal based on the first specular blending mask to obtain an intermediate blended image.

[0214] In some exemplary embodiments of this disclosure, the first specular fusion unit 1504 is configured to: obtain a first specular fusion mask based on the overexposed areas of the deghosted reference image; and perform specular fusion on the deghosted reference image and the non-reference image based on the first specular fusion mask to obtain an intermediate fused image.

[0215] In some exemplary embodiments of this disclosure, the second highlight fusion unit 1505 is configured to: obtain a second highlight fusion mask based on the overexposed areas of the intermediate fusion image; and perform highlight fusion of the intermediate fusion image and the third aligned image based on the second highlight fusion mask to obtain an HDR image.

[0216] In some exemplary embodiments of this disclosure, the second specular fusion unit 1505 is configured to: perform brightness comparison on the intermediate fused image and the third aligned image according to the second specular fusion mask to determine a brightness abnormal region; perform brightness correction on the third aligned image in response to the area of ​​the brightness abnormal region exceeding an area threshold; and perform specular fusion on the intermediate fused image and the brightness-corrected third aligned image according to the second specular fusion mask to obtain an HDR image.

[0217] In some exemplary embodiments of this disclosure, the second specular fusion unit 1505 is configured to: generate a specular fusion boundary mask based on the second specular fusion mask; perform brightness comparison on the intermediate fused image and the third aligned image based on the specular fusion boundary mask, and calculate the boundary brightness gradient; and determine a brightness abnormal region in response to the boundary brightness gradient exceeding a gradient threshold.

[0218] In some exemplary embodiments of this disclosure, the second specular fusion unit 1505 is configured to: perform brightness comparison pixel by pixel on the intermediate fused image and the third aligned image to obtain a correction weight mask; and perform brightness correction on the third aligned image based on the second specular fusion mask and the correction weight mask.

[0219] In some exemplary embodiments of this disclosure, the second highlight fusion unit 1505 is configured to: obtain a highlight white mask based on the overexposed areas of the third aligned image; and perform brightness correction on the third aligned image based on the second highlight fusion mask, the highlight white mask, and the correction weight mask.

[0220] In some exemplary embodiments of this disclosure, the second highlight fusion unit 1505 is further configured to feather the correction weight mask.

[0221] In some exemplary embodiments of this disclosure, the provided image processing apparatus further includes: a highlight anomaly processing unit, configured to:

[0222] Detect highlight anomalies in intermediate fused images and / or HDR images to identify highlight anomaly regions;

[0223] Image retouching is performed on areas of abnormal highlights.

[0224] In some exemplary embodiments of this disclosure, the highlight anomaly processing unit is configured to: perform image retouching on the highlight anomaly region in response to the highlight anomaly region not exceeding the highlight anomaly threshold; and retain the image corresponding to the intermediate fused image for the highlight anomaly region in response to the highlight anomaly region exceeding the highlight anomaly threshold.

[0225] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0226] Figure 16 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. For example, device 1600 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0227] Reference Figure 16 The device 1600 may include one or more of the following components: a processing component 1602, a memory 1604, a power supply component 1606, a multimedia component 1608, an audio component 1610, an input / output (I / O) interface 1612, a sensor component 1614, and a communication component 1616.

[0228] Processing component 1602 typically controls the overall operation of device 1600, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1602 may include one or more processors 1620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1602 may include one or more modules to facilitate interaction between processing component 1602 and other components. For example, processing component 1602 may include a multimedia module to facilitate interaction between multimedia component 1608 and processing component 1602.

[0229] Memory 1604 is configured to store various types of data to support the operation of device 1600. Examples of this data include instructions for any application or method operating on device 1600, contact data, phonebook data, messages, pictures, videos, etc. Memory 1604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0230] Power supply component 1606 provides power to various components of device 1600. Power supply component 1606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 1600.

[0231] Multimedia component 1608 includes a screen that provides an output interface between the device 1600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1608 includes a front-facing camera and / or a rear-facing camera. When the device 1600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0232] Audio component 1610 is configured to output and / or input audio signals. For example, audio component 1610 includes a microphone (MIC) configured to receive external audio signals when device 1600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1604 or transmitted via communication component 1616. In some embodiments, audio component 1610 also includes a speaker for outputting audio signals.

[0233] I / O interface 1612 provides an interface between processing component 1602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0234] Sensor assembly 1614 includes one or more sensors for providing status assessments of various aspects of device 1600. For example, sensor assembly 1614 may detect the on / off state of device 1600, the relative positioning of components such as the display and keypad of device 1600, changes in position of device 1600 or a component of device 1600, the presence or absence of user contact with device 1600, the orientation or acceleration / deceleration of device 1600, and temperature changes of device 1600. Sensor assembly 1614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1614 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0235] Communication component 1616 is configured to facilitate wired or wireless communication between device 1600 and other devices. Device 1600 can access wireless networks based on communication standards such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 1616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 1616 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0236] In some embodiments of this disclosure, the apparatus 1600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0237] In some embodiments of this disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1604 including instructions, which can be executed by a processor 1620 of device 1600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0238] In some embodiments of this disclosure, a non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform an image processing method, the method comprising:

[0239] Receive a first image, a second image, and a third image; wherein the exposure time of the first image is greater than that of the second image, and the exposure time of the second image is greater than that of the third image;

[0240] The first image, the second image, and the third image are brightness aligned to obtain the brightness-aligned first aligned image, the second aligned image, and the third aligned image.

[0241] Ghosting removal is performed on the first and second aligned images;

[0242] A specular blending is performed on the first and second aligned images after ghosting removal to obtain an intermediate blended image.

[0243] High dynamic range (HDR) images are obtained by performing specular fusion based on the intermediate fused image and the third aligned image.

[0244] In some embodiments of this disclosure, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement an image processing method, the method comprising:

[0245] Receive a first image, a second image, and a third image; wherein the exposure time of the first image is greater than that of the second image, and the exposure time of the second image is greater than that of the third image;

[0246] The first image, the second image, and the third image are brightness aligned to obtain the brightness-aligned first aligned image, the second aligned image, and the third aligned image.

[0247] Ghosting removal is performed on the first and second aligned images;

[0248] A specular blending is performed on the first and second aligned images after ghosting removal to obtain an intermediate blended image.

[0249] High dynamic range (HDR) images are obtained by performing specular fusion based on the intermediate fused image and the third aligned image.

[0250] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0251] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Receive a first image, a second image, and a third image; wherein the exposure duration of the first image is greater than that of the second image, and the exposure duration of the second image is greater than that of the third image; The first image, the second image, and the third image are brightness aligned to obtain a brightness-aligned first image, a second image, and a third image. Ghosting removal is performed on the first aligned image and the second aligned image; A specular blending is performed on the first and second aligned images after ghosting removal to obtain an intermediate blended image; High dynamic range (HDR) images are obtained by performing specular fusion based on the intermediate fused image and the third aligned image.

2. The image processing method according to claim 1, characterized in that, Ghosting removal is performed on the first and second aligned images, including: The first aligned image and the second aligned image are input into a pre-trained ghost removal model to obtain the first aligned image and the second aligned image after ghost removal processing.

3. The image processing method according to claim 2, characterized in that, The ghosting removal model is a student model obtained by using network distillation based on a pre-trained teacher model.

4. The image processing method according to claim 2, characterized in that, The first aligned image and the second aligned image are input into a pre-trained ghosting removal model to obtain the first aligned image and the second aligned image after ghosting removal, including: Determine the reference image in the first and second aligned images; The first aligned image, the second aligned image, and the reference image determination information are input into the ghost removal model to obtain the reference image after ghost removal processing.

5. The image processing method according to claim 4, characterized in that, Determining a reference image in the first and second aligned images includes: Determine the proportion of ghosting in the first aligned image; In response to the ghost image proportion being less than or equal to a preset proportion threshold, the first aligned image is determined to be the reference image; In response to the ghost image ratio being greater than a preset ratio threshold, the second aligned image is determined as the reference image.

6. The image processing method according to claim 1, characterized in that, Spectral fusion is performed on the first and second aligned images after ghosting removal to obtain an intermediate fused image, including: Based on the overexposed areas of the first aligned image after ghosting removal, a first specular blending mask is obtained; Based on the first specular blending mask, the first aligned image and the second aligned image after ghosting removal are specularly blended to obtain the intermediate blended image.

7. The image processing method according to claim 4, characterized in that, Spectral fusion is performed on the first and second aligned images after ghosting removal to obtain an intermediate fused image, including: Based on the overexposed areas of the reference image after ghosting removal, a first specular blending mask is obtained; Based on the first specular fusion mask, the reference image and the non-reference image after ghosting removal are specularly fused to obtain the intermediate fused image.

8. The image processing method according to claim 1, characterized in that, Highlight fusion is performed based on the intermediate fused image and the third aligned image to obtain an HDR image, including: Based on the overexposed areas of the intermediate fused image, a second highlight fusion mask is obtained; Based on the second specular fusion mask, the intermediate fused image and the third aligned image are specularly fused to obtain the HDR image.

9. The image processing method according to claim 8, characterized in that, Based on the second specular fusion mask, the intermediate fused image and the third aligned image are specularly fused to obtain the HDR image, including: Based on the second specular fusion mask, the brightness of the intermediate fused image and the third aligned image are compared to determine the areas of abnormal brightness. In response to the area of ​​the abnormal brightness region exceeding the area threshold, brightness correction is performed on the third aligned image; Based on the second specular blending mask, the intermediate blended image and the brightness-corrected third aligned image are specularly blended to obtain the HDR image.

10. The image processing method according to claim 9, characterized in that, Based on the second specular blending mask, a brightness comparison is performed between the intermediate blended image and the third aligned image to determine areas of abnormal brightness, including: Based on the second specular fusion mask, a specular fusion boundary mask is generated; Based on the aforementioned specular fusion boundary mask, the brightness of the intermediate fused image and the third aligned image are compared, and the boundary brightness gradient is calculated. In response to the boundary brightness gradient exceeding a gradient threshold, the brightness anomaly region is identified.

11. The image processing method according to claim 9, characterized in that, Brightness correction is performed on the third aligned image, including: The brightness of each pixel in the intermediate fused image and the third aligned image is compared to obtain a correction weight mask. The brightness of the third aligned image is corrected based on the second specular fusion mask and the correction weight mask.

12. The image processing method according to claim 11, characterized in that, Based on the second specular fusion mask and the correction weight mask, brightness correction is performed on the third aligned image, including: Based on the overexposed areas of the third aligned image, a highlight white mask is obtained; The brightness of the third aligned image is corrected based on the second specular blending mask, the specular white mask, and the correction weight mask.

13. The image processing method according to claim 11, characterized in that, The brightness of each pixel in the intermediate fused image and the third aligned image is compared to obtain a corrected weight mask, and the method further includes: The correction weight mask is feathered.

14. The image processing method according to claim 1, characterized in that, Also includes: Detect highlight anomalies in the intermediate fused image and / or the HDR image to identify highlight anomaly regions; Image retouching is performed on the aforementioned areas of abnormal highlight.

15. The image processing method according to claim 14, characterized in that, Image retouching of the aforementioned highlight anomaly region includes: In response to the fact that the highlight anomaly region does not exceed the highlight anomaly threshold, image retouching is performed on the highlight anomaly region; In response to the highlight anomaly region exceeding the highlight anomaly threshold, the image corresponding to the intermediate fused image is retained for the highlight anomaly region.

16. The image processing method according to claim 1, characterized in that, Receive the first image, the second image, and the third image, including: Preprocess the first image, the second image, and the third image; Receive the preprocessed first image, second image, and third image; The preprocessing includes at least one of the following: noise reduction and motion alignment.

17. An image processing apparatus, characterized in that, include: An image receiving unit is configured to receive a first image, a second image, and a third image; wherein the exposure time of the first image is greater than that of the second image, and the exposure time of the second image is greater than that of the third image; A brightness alignment unit is used to perform brightness alignment on the first image, the second image, and the third image to obtain a brightness-aligned first aligned image, a second aligned image, and a third aligned image. The ghosting removal unit is used to perform ghosting removal processing on the first aligned image and the second aligned image. The first specular fusion unit is used to perform specular fusion based on the first aligned image and the second aligned image after ghosting removal to obtain an intermediate fused image. The second specular fusion unit is used to perform specular fusion based on the intermediate fused image and the third aligned image to obtain a high dynamic range (HDR) image.

18. An image processing apparatus, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the image processing method according to any one of claims 1 to 16.

19. A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to perform an image processing method according to any one of claims 1 to 16.

Images