Image Fusion Method, Device, Computer-Readable Medium, and Electronic Device

Abstract

The present disclosure provides an image fusion method, an apparatus, a computer-readable medium, and an electronic device, relating to the technical field of image capture. The method includes: obtaining an original exposure image, where the original exposure image includes a first exposure image belonging to the same exposure parameter and a second exposure image belonging to different exposure parameters; performing registration and alignment processing on the original exposure image to obtain the registered and aligned original exposure image; and performing image fusion on the registered and aligned original exposure image to generate a target output image. By registering and aligning the original exposure images collected at different exposure times and then performing image fusion, the present disclosure can effectively reduce the ghosting phenomenon in the generated target output image, improve the clarity of the texture details in the target output image, and enhance the display quality of the target output image.

Description

Background Art

[0002] High Dynamic Range (HDR) imaging technology is a set of technologies used to achieve a greater exposure dynamic range (i.e., a greater difference between light and dark) than ordinary digital image technologies. "Dynamic range" is a term used to define the range within which a camera can capture tonal details of an image, usually referring to the range between the lowest value and the highest overflow value. Simply put, it describes the ratio between the brightest and darkest tones that a camera can record in a single frame. The greater the dynamic range, the more information in these highlight and shadow areas can be retained as much as possible.

[0003] Currently, in the related art, when fusing multiple exposure images, "ghosting" is likely to occur, that is, the phenomenon of image smear, resulting in the loss of image texture details and poor image quality. Summary of the Invention

[0004] The purpose of the present disclosure is to provide an image fusion method, an image fusion device, a computer-readable medium, and an electronic device, thereby at least to a certain extent reducing the smear phenomenon in the target output image, enhancing the clarity of texture details, and improving the image quality of the target output image.

[0005] According to the first aspect of the present disclosure, there is provided an image fusion method, including:

[0006] Obtaining original exposure images, where the original exposure images include a first exposure image belonging to the same exposure parameter and a second exposure image belonging to different exposure parameters;

[0007] Performing registration and alignment processing on the original exposure images to obtain the registered and aligned original exposure images;

[0008] Performing image fusion on the registered and aligned original exposure images to generate a target output image.

[0009] According to the second aspect of the present disclosure, there is provided an image fusion device, including:

[0010] An image acquisition module, configured to obtain original exposure images, where the original exposure images include a first exposure image belonging to the same exposure parameter and a second exposure image belonging to different exposure parameters;

[0011] An image alignment module, configured to perform registration and alignment processing on the original exposure images to obtain the registered and aligned original exposure images;

[0012] An image fusion module, configured to perform image fusion on the registered and aligned original exposure images to generate a target output image.

[0013] According to a third aspect of the present disclosure, there is provided a computer-readable medium having stored thereon a computer program, which when executed by a processor implements the above-described method.

[0014] According to a fourth aspect of the present disclosure, there is provided an electronic device, characterized by comprising:

[0015] a processor; and

[0016] a memory for storing one or more programs, which when executed by one or more processors cause the one or more processors to implement the above-described method.

[0017] The image fusion method provided by an embodiment of the present disclosure may first perform registration alignment processing on the acquired original exposure images to obtain the registered and aligned original exposure images, and then may perform image fusion on the registered and aligned original exposure images to generate a target output image. Performing registration alignment on the original exposure images before image fusion can ensure the accuracy of the image content expression at the same positions in the original exposure images, effectively reduce the ghosting phenomenon in the target output image obtained after image fusion, thereby improving the clarity of texture details and the image quality of the target output image.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts. In the drawings:

[0020] Figure 1 Schematically shows a schematic diagram of the stage when the image fusion method in an exemplary embodiment of the present disclosure is applied;

[0021] Figure 2 Schematically shows a flowchart of an image fusion method in an exemplary embodiment of the present disclosure;

[0022] Figure 3 Schematically shows a flowchart of performing registration alignment on an original exposure image in an exemplary embodiment of the present disclosure;

[0023] Figure 4 Schematically shows a schematic diagram of the principle of statistical luminance information in an exemplary embodiment of the present disclosure;

[0024] Figure 5 Schematically show the schematic diagram of the principle of remapping through a local image transformation matrix in an exemplary embodiment of the present disclosure;

[0025] Figure 6 Schematically show the schematic diagram of the process of image fusion for the original exposure image after registration and alignment in an exemplary embodiment of the present disclosure;

[0026] Figure 7 Schematically show the schematic diagram of the process of image fusion for the first exposure image in an exemplary embodiment of the present disclosure;

[0027] Figure 8 Schematically show the schematic diagram of the principle of image fusion based on the type of image block in an exemplary embodiment of the present disclosure;

[0028] Figure 9 Schematically show the schematic diagram of a mapping curve in an exemplary embodiment of the present disclosure;

[0029] Figure 10 Schematically show the schematic diagram of the composition of the image fusion device in an exemplary embodiment of the present disclosure;

[0030] Figure 11 Show the schematic diagram of an electronic device to which the embodiments of the present disclosure can be applied. Detailed implementation manners

[0031] Now, the exemplary embodiments will be described more comprehensively with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more complete and comprehensive, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0032] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0033] HDR technology can be used to merge several images with different exposure values (EV) together to recover the highlight and shadow details in a high-contrast environment. For example, in a sunset shooting environment, a single shot will generally result in underexposure or overexposure in the output image; using multiple exposures and HDR synthesis in post-production, you can get a photo with details in both bright and dark areas. Although HDR can record the details of highlights and shadows, if too much of these details are restored in post-production, it will cause image distortion. Therefore, the key to using HDR technology to obtain high dynamic range images is how to choose the appropriate brightness and details when fusing multiple exposure images.

[0034] In related technologies, HDR technology generally acts on images in the RGB color space that have undergone the digital image signal processing (Image Single Process, ISP) process. However, when performing high dynamic range synthesis on RGB color space images, the amount of calculation is larger than that on Raw domain data, and the real-time performance is poor. In addition, when performing HDR synthesis on multiple exposure images, related technologies directly perform image fusion on the multiple exposure images. This solution is more prone to ghosting, resulting in loss of image texture details and poor image quality.

[0035] In view of this, the present disclosure first provides a new image fusion method to reduce the smear phenomenon in the image obtained after image fusion, thereby improving the clarity of image texture details and improving the image quality of the output image.

[0036] The image fusion method of the embodiments of the present disclosure can be implemented by an electronic device. That is, the electronic device can perform each step of the image fusion method described below, and the image fusion device described below can be configured within the electronic device. For example, the image processing scheme of the present disclosure can be implemented by an image signal processor equipped in the electronic device. In addition, the present disclosure does not limit the type of electronic device, which may include but is not limited to smartphones, tablets, smart wearable devices, personal computers, servers, etc.

[0037] Optionally, the image fusion method of the embodiment of the present disclosure may also be implemented by a server or a server cluster, and the corresponding image fusion device may be provided within the server or the server cluster. Of course, the image fusion method of the embodiment of the present disclosure may also be performed by a system consisting of an electronic device and a server or a server cluster. For example, the electronic device may capture the original exposure image and then upload it to the server or the server cluster. After the server or the server cluster generates the target output image through the image fusion method of the embodiment of the present disclosure, the target output image is returned to the electronic device for storage, display, and post-processing. This example embodiment does not specifically limit the manner in which the image fusion method is performed.

[0038] Figure 1 A schematic diagram showing the stage at which the image fusion method in an exemplary embodiment of the present disclosure is applied. Refer to Figure 1 As shown, in the implementation manner of the present disclosure, the input image of the image fusion method may be the image sensor 120 in the camera module of the electronic device 110, and the original exposure images 130 collected at different exposure times (which can also be understood as exposure value EV). The original exposure images 130 may be image data in the Raw domain. The original exposure images 130 may be input into an image signal processor, and the image signal processor may generate a target output image 150 through the image fusion method provided by the exemplary embodiment of the present disclosure in the image signal processing pipeline 140 (ISP Pipeline). Of course, Figure 1 in this example, the original exposure images 130 are taken as the real-time acquisition by the electronic device 110 for illustration. The original exposure images 130 may also be the original exposure images obtained by the electronic device from the outside (i.e., other devices). The present disclosure places no restrictions on the image source, image content, image size, etc. of the original exposure images.

[0039] Taking the electronic device executing this method as an example below, the image fusion method and the image fusion device of the exemplary embodiment of the present disclosure will be specifically described.

[0040] Figure 2 The flowchart of an image fusion method in this exemplary embodiment is shown, which may include the following steps S210 to step S230:

[0041] In step S210, obtain the original exposure images, where the original exposure images include a first exposure image belonging to the same exposure parameter and a second exposure image belonging to different exposure parameters.

[0042] In an exemplary embodiment, the original exposure images may be multiple frames of image data with different exposure parameters. For example, the exposure parameter may be the exposure time, and the original exposure images may be the image data collected at different exposure times; of course, the exposure parameter may also be the exposure value or the aperture light input amount, and the original exposure images may be the image data collected at different exposure values or aperture light input amounts. This exemplary embodiment places no special limitation on the type of the exposure parameter for controlling the light input amount.

[0043] It can be understood that the original exposure images may include exposure images under different exposure parameters. Additionally, under the same exposure parameter, one frame of exposure image may be collected, or multiple frames of exposure images may be collected, that is, the original exposure images may also include at least one frame of exposure image belonging to the same exposure parameter.

[0044] The original exposure image can be the Raw domain image data captured in real time by an image sensor. It is easy to understand that the original exposure image can also be the RGB domain image data or YUV domain image data generated through an image signal processing flow, etc. Moreover, the image bit width of the input original exposure image can be 8 bits or 10 bits. Of course, it can also be the original exposure image with other bit widths, and this exemplary embodiment does not make special limitations on this.

[0045] The original exposure image can be captured in real time by the camera module of an electronic device, or can be obtained from other electronic devices in a wireless communication manner or a wired communication manner. This embodiment does not make special limitations on the source of the original exposure image.

[0046] In step S220, perform registration and alignment processing on the original exposure image to obtain the registered and aligned original exposure image.

[0047] In an exemplary embodiment, the registration and alignment processing refers to the processing process of aligning the image content of the original exposure images under different exposure parameters. For example, the registration and alignment processing can be to perform brightness alignment processing on the original exposure images under different exposure parameters to unify the brightness of the original exposure images under different exposure parameters to the same level; the registration and alignment processing can also be to perform image texture alignment processing on different original exposure images to align the image textures in the original exposure images with the same image content spatially. Of course, other types of alignment methods can also be used to perform registration and alignment on multiple frames of original exposure images. For example, determine the region of interest (such as the face region) in the original exposure image, and then the multiple frames of original exposure images can be aligned according to the region of interest. This exemplary embodiment does not make any special limitations on this.

[0048] In step S230, perform image fusion on the registered and aligned original exposure image to generate a target output image.

[0049] In an exemplary embodiment, after registering and aligning the original exposure images under different exposure parameters, perform image fusion on the registered and aligned original exposure images. Optionally, image fusion can be performed on the exposure images belonging to the same exposure parameter, and then image fusion can be performed on the exposure images belonging to different exposure parameters; of course, it is also possible not to fuse separately by exposure parameters, but directly perform pairwise fusion on the registered and aligned original exposure images. This exemplary embodiment does not make special limitations on the image fusion method of the registered and aligned original exposure images.

[0050] First, the acquired original exposure images can be subjected to registration and alignment processing to obtain the registered and aligned original exposure images. Then, the registered and aligned original exposure images can be subjected to image fusion, which can ensure the accuracy of the image content expression at the same position in the original exposure images, effectively reduce the smear phenomenon in the target output image obtained after image fusion, thereby enhancing the clarity of texture details and improving the image quality of the target output image.

[0051] The following will elaborate on steps S210 to S230 in detail.

[0052] In an exemplary embodiment, the original exposure images may include a first exposure image belonging to the same exposure parameter and a second exposure image belonging to different exposure parameters. Of course, the "first" and "second" in the "first exposure image" and "second exposure image" here are only used to distinguish the original exposure images belonging to the same or different exposure parameters, without any special meaning and should not impose any special limitations on this exemplary embodiment.

[0053] For example, the exposure parameter can be the exposure level. The exposure level combination list for acquiring the original exposure images can be {EV0, EV0, EV0, EV+2, EV-2}, that is, control the camera module to acquire 3 frames of original exposure images at the exposure level EV0, control the camera module to acquire 1 frame of original exposure image at the exposure level EV+2, and control the camera module to acquire 1 frame of original exposure image at the exposure level EV-2. The 3 frames of original exposure images acquired at the exposure level EV0 belong to the first exposure images of the same exposure parameter, and the exposure images at the EV0, EV+2, and EV-2 levels respectively belong to the second exposure images of different exposure parameters.

[0054] Of course, multiple frames of exposure images can be acquired under different exposure parameters. For example, multiple frames of original exposure images can also be acquired at the EV+2 and EV-2 levels. For example, 4 or 5 frames of original exposure images can be acquired at the EV0 level, 4 or 5 frames of original exposure images can be acquired at the EV+2 level, and 4 or 5 frames of original exposure images can be acquired at the EV-2 level. This exemplary embodiment does not make any special limitations in this regard.

[0055] Optionally, the image brightness relationship of the exposure levels EV0, EV+2, and EV-2 in each gear in this embodiment can conform to {EV+2: EV0: EV-2 = 4:1:0.25}. Of course, other exposure brightness combinations can also be adopted for the image brightness relationship of each gear, and this exemplary embodiment is not limited thereto.

[0056] It can be understood that the exposure list and exposure levels in this embodiment are illustrative. For example, the exposure list can also be {EV0, EV0, EV0, EV-1, EV-2}, or {EV+1, EV0, EV0, EV-1, EV-2}. This exemplary embodiment does not impose any special restrictions on the exposure combination.

[0057] For ease of explanation, subsequent embodiments will use the original exposure images collected when the exposure combination list is {EV0, EV0, EV0, EV+2, EV-2} as examples for illustration.

[0058] In an exemplary embodiment, the registration and alignment processing of the original exposure image can be achieved through the steps in Figure 3 , referring to Figure 3 shown, and specifically may include:

[0059] Step S310, performing brightness alignment processing on the second exposure image to obtain the second exposure image with brightness alignment;

[0060] Step S320, performing image texture alignment on the first exposure image and the second exposure image with brightness alignment to obtain the target exposure image.

[0061] Among them, the second exposure image is the original exposure image belonging to different exposure parameters. The brightness alignment processing can be performed on the second exposure image to unify the brightness levels of the original exposure images under different exposure parameters to the same level, improving the accuracy of the brightness information in the original exposure images under different exposure parameters. For example, the second exposure image can include the original exposure image in the EV0 gear, the original exposure image in the EV+2 gear, and the original exposure image in the EV-2 gear. The brightness ratio relationship of the original exposure images under different exposure degrees can be determined, and then the brightness information of the original exposure images under different exposure degrees can be mapped to the same brightness level through this brightness ratio relationship.

[0062] Usually, when the camera exposes, it will transmit the exposure information to the algorithm side in the form of metadata for use. However, in actual camera exposure, the situation of inaccurate exposure ratio often occurs. Under inaccurate exposure ratio, the brightness relationship matching of the image goes wrong, which will affect the subsequent motion detection and image fusion effects. Therefore, by performing brightness alignment processing on the second exposure images belonging to different exposure parameters, the accuracy of the exposure ratio of the exposure images under different exposure parameters can be effectively guaranteed, the accuracy of the brightness information in the original exposure images under different exposure parameters can be improved, and thus the accuracy of the subsequent image fusion result can be ensured.

[0063] After obtaining the second exposure image with brightness alignment, the first exposure image belonging to the same exposure parameter and the second exposure image with brightness alignment can be subjected to image texture alignment to ensure the accuracy of the expression of the image content at the same position in the first exposure image and the second exposure image, ensure that there is no ghosting phenomenon in the fusion result obtained by fusing the first exposure image and the second exposure image with brightness alignment, ensure the clarity of the texture details in the image content, and improve the image quality. For example, image texture alignment can be achieved by detecting the image feature points in each exposure image and remapping the pixel points in different exposure images by determining the image feature point pairs; of course, image texture alignment can also be achieved by determining the region of interest in each exposure image, determining the alignment parameters of the region of interest, and then performing image texture alignment on the first exposure image and the second exposure image with brightness alignment according to the alignment parameters. The embodiments of this example do not make any special limitations on the methods for achieving image texture alignment.

[0064] In an exemplary embodiment, the brightness alignment process of the second exposure image can be achieved through the following steps: the brightness information corresponding to the second exposure image can be counted; the brightness mapping data between the second exposure images can be determined according to the brightness information; and the second exposure image can be subjected to brightness alignment processing based on the brightness mapping data to obtain the second exposure image with brightness alignment.

[0065] Among them, the brightness information refers to the information related to the image brightness included in the second exposure image. For example, the brightness value corresponding to each pixel in the second exposure image can be counted by the histogram statistics method to obtain the image brightness histogram distribution map, and this image brightness histogram distribution map can be used as the brightness information corresponding to the second exposure image; or the brightness information corresponding to the second exposure image can be determined by other methods capable of calculating the image brightness. The embodiments of this example do not make any special limitations on this.

[0066] The brightness mapping data can be a mapping relationship for balancing the exposure ratio or brightness ratio between the second exposure images calculated according to the brightness information. For example, the second exposure image can include the original exposure image in the EV0 gear, the original exposure image in the EV+2 gear, and the original exposure image in the EV-2 gear. The brightness mapping data can include the brightness factor A between the original exposure image in the EV+2 gear and the original exposure image in the EV0 gear, and the brightness factor B between the original exposure image in the EV-2 gear and the original exposure image in the EV0 gear. Among them, the calculation process of the brightness factor A and the brightness factor B can be represented by the relational expression (1):

[0067]

[0068] Among them, A can represent the brightness factor between the original exposure image of EV + 2 stops and the original exposure image of EV0 stop, B can represent the brightness factor between the original exposure image of EV - 2 stops and the original exposure image of EV0 stop, R EV+2 can represent the brightness information of the original exposure image of EV + 2 stops, R EV0 can represent the brightness information of the original exposure image of EV0 stop, R EV-2 can represent the brightness information of the original exposure image of EV - 2 stops.

[0069] After determining the brightness mapping data, the second exposure image can be subjected to brightness alignment processing based on the brightness mapping data. For example, the correspondence between the original exposure image at EV0 stop, the original exposure image at EV + 2 stops, and the original exposure image at EV - 2 stops can be determined according to the brightness mapping data, and this correspondence can be represented by formula (2):

[0070]

[0071] Among them, A can represent the brightness factor between the original exposure image of EV + 2 stops and the original exposure image of EV0 stop, B can represent the brightness factor between the original exposure image of EV - 2 stops and the original exposure image of EV0 stop, P EV+2 can represent the pixel point data of the original exposure image of EV + 2 stops, P EV0 can represent the pixel point data of the original exposure image of EV0 stop, P EV-2 can represent the pixel point data of the original exposure image of EV - 2 stops. Optionally, any exposure parameter can be selected as the reference brightness image. For example, the original exposure image of EV0 stop is used as the reference brightness image, and then through the pixel point data P of the original exposure image of EV0 stop EV0 the brightness factor A, and the brightness factor B, the updated pixel point data in the original exposure images of EV - 2 stops and EV + 2 stops are calculated to achieve the brightness alignment of the second exposure image.

[0072] By performing brightness alignment processing on the second exposure images belonging to different exposure parameters, the accuracy of the exposure ratio of the exposure images under different exposure parameters can be effectively guaranteed, the accuracy of the brightness information in the original exposure images under different exposure parameters can be improved, and thus the accuracy of the subsequent image fusion result can be guaranteed.

[0073] Optionally, in this embodiment, the brightness information corresponding to the second exposure image can be statistically calculated through the following steps: the preset number of pixel blocks can be obtained, and the pixel blocks in the second exposure image can be divided according to the number of pixel blocks; the target pixel blocks in the divided second exposure image can be determined; the brightness histogram of the target pixel blocks can be calculated, and the brightness information corresponding to the second exposure image can be determined according to the brightness histogram.

[0074] Among them, the pixel blocks in the second exposure image refer to the pixel units composed of different color channels in the exposure image. For example, if the second exposure image is a Raw domain exposure image collected by an image sensor based on the RGBG filter pattern, then a pixel block can be a pixel unit composed of an RGBG block; the second exposure image can also be a Raw domain exposure image collected by an image sensor based on the RYYB filter pattern, then a pixel block can be a pixel unit composed of an RYYB block. This exemplary embodiment does not make special limitations on this.

[0075] The number of pixel blocks refers to the number preset for statistically calculating the brightness information. For example, if the number of pixel blocks is 4, then 4 pixel blocks (such as RGBG blocks) in the second exposure image can be used as a minimum statistical unit to statistically calculate the brightness information in the second exposure image.

[0076] The target pixel blocks refer to the pixel blocks selected from the divided pixel blocks for statistically calculating the brightness information in the second exposure image.

[0077] The target pixel blocks can be determined by means of skip-block statistics. For example, multiple 4*4 pixel blocks (4*RGBG blocks) can be obtained by dividing according to the number of pixel blocks, and then skip-block statistics can be performed on these 4*4 pixel blocks. For example, the skip-block step size can be 1, that is, every time a 4*4 pixel block is skipped, the next 4*4 pixel block is used as the target pixel block. Of course, the skip-block step size can also be 2, 3, etc. This exemplary embodiment does not make special limitations on this.

[0078] The target pixel blocks can also be determined by detecting overexposed points. For example, multiple 4*4 pixel blocks (4*RGBG blocks) can be obtained by dividing according to the number of pixel blocks, and then it can be detected whether there are overexposed points in these 4*4 pixel blocks. The 4*4 pixel blocks with overexposed points are excluded and do not participate in the statistical calculation of the brightness information. After traversing all 4*4 pixel blocks, the target pixel blocks are obtained.

[0079] The target pixel blocks can also be obtained by further screening for overexposed points on the pixel blocks determined by the skip-block statistics method. For example, after some pixel blocks are determined by skip-block statistics, it can be detected whether there are overexposed points in these pixel blocks, and the pixel blocks with overexposed points are excluded to obtain a smaller number of target pixel blocks.

[0080] By block skipping statistics, the number of target pixel blocks can be effectively reduced, the amount of statistics of luminance information can be decreased, and the statistical efficiency of luminance information can be improved; by overexposed point screening, pixel blocks that may affect the accuracy of luminance information can be effectively reduced, and the accuracy of the statistically obtained luminance information can be improved.

[0081] Figure 4 Schematically shows a schematic diagram of the principle of statistically obtaining luminance information in an exemplary embodiment of the present disclosure.

[0082] As shown in reference Figure 4 For the second exposure image 410, according to a preset number of pixel blocks, for example, the number of pixel blocks can be 4, that is, 4 RGBG blocks can be used as a pixel block 420, and 16 pixel blocks can be obtained. Optionally, the block skipping step size can be set to 1, and the target pixel blocks (such as the pixel blocks corresponding to the thick line frames) can be determined among these 16 pixel blocks by block skipping statistics, and 8 target pixel blocks can be determined, effectively reducing the amount of statistics of luminance information and improving the statistical efficiency of luminance information.

[0083] Optionally, in addition to using the statistically obtained luminance histogram as the luminance information corresponding to the second exposure image, in this embodiment, the luminance information corresponding to the second exposure image can be determined according to the luminance histogram through the following steps: a preset luminance interval can be obtained, and target luminance data can be screened in the luminance histogram according to the luminance interval; the luminance information corresponding to the second exposure image can be determined based on the target luminance data.

[0084] Among them, the luminance interval refers to the data preset for screening the effective luminance information in the second exposure image. For example, the luminance interval can be 5%-15% (non-saturated dark area), that is, the luminance data within the 5%-15% interval in the luminance histogram can be used as the target luminance data. It can be understood that the statistically obtained luminance data can fall within the same luminance histogram. For example, the luminance histogram can be in the range of 0-255, and the luminance information within the interval of 255*5%-255*15% can be taken as the target luminance data, and then the target luminance data can be used as the luminance information corresponding to the second exposure image. The luminance interval can also be 10%-20%, and can be specifically set customarily according to the actual application situation, and this exemplary embodiment does not make special limitations on this.

[0085] By screening the luminance data in the luminance histogram through the luminance interval, the amount of data participating in subsequent calculations can be effectively reduced, and the statistical efficiency of luminance information can be improved; at the same time, by smoothing the luminance information in the second exposure image through the luminance interval, the accuracy of the luminance information can be further improved.

[0086] In an exemplary embodiment, the image texture alignment of the first exposure image and the second exposure image after brightness alignment can be achieved through the following steps to obtain a target exposure image: A reference image can be selected from the first exposure image or the second exposure image after brightness alignment, and the exposure images other than the reference image in the first exposure image and the second exposure image after brightness alignment are used as the images to be registered; Feature point pairs between the reference image and the images to be registered are determined, and an image transformation matrix is determined based on the feature point pairs; Pixel points in the images to be registered are remapped according to the image transformation matrix to obtain a target exposure image.

[0087] The reference image refers to an exposure image with relatively accurate image content expression in the first exposure image and the second exposure image after brightness alignment. For example, for 1 frame of original exposure image collected at EV+2, 3 frames of original exposure image collected at EV0, and 1 frame of original exposure image collected at EV-2, among these 5 frames of original exposure images, the 3rd frame of original exposure image collected at EV0 can be selected as the reference image; The image sharpness of the first exposure image and the second exposure image after brightness alignment can also be determined, and the exposure image with the highest image sharpness (usually, image sharpness is characterized by image sharpness, and the clearest frame means less motion blur) is selected as the reference image; The image gradients of the first exposure image and the second exposure image after brightness alignment can also be calculated, and the exposure image with the largest image gradient (the larger the image gradient, the clearer the image texture) is selected as the reference image. Of course, the reference image can also be determined through other screening methods, which can be specifically customized according to the actual usage scenario, and this exemplary embodiment does not make special limitations on this.

[0088] The feature point pair refers to a point pair composed of image feature points between the reference image and the images to be registered. For example, the feature points in the reference image and the images to be registered can be detected through the Speeded Up Robust Features (SURF) operator; The feature points in the reference image and the images to be registered can also be detected through the Scale-invariant feature transform (SIFT) operator. Of course, other methods can also be used to determine the image feature points in the reference image and the images to be registered, and this exemplary embodiment does not make special limitations on this.

[0089] The pixel values in the reference image and the images to be registered can be averaged to obtain the grayscale images of the reference image and the images to be registered, and the feature point pairs between the reference image and the images to be registered can be determined in the grayscale images, which can effectively improve the search efficiency of image feature points and the accuracy of image feature points.

[0090] Image feature points can be determined in the reference image first. For example, the SURF operator can be used to determine image feature points in the reference image. Then, for the position of each detected image feature point in the reference image, a quick search and match is performed at the same position in the image to be registered. The quick search converges to the best matching position, and the matching feature point pairs are calculated.

[0091] After determining the feature point pairs between the reference image and the image to be registered, the image transformation matrix (Homography) can be calculated based on all the feature point pairs. For example, the image transformation matrix can be calculated by the Random Sample Consensus algorithm (Ransac). Furthermore, the pixel points in the image to be registered can be remapped through the image transformation matrix to obtain the target exposure image.

[0092] Optionally, the image transformation matrix can include a global image transformation matrix and a local image transformation matrix. The following steps can be used to determine the image transformation matrix based on the feature point pairs: A preset division ratio can be obtained, and the reference image and the image to be registered are divided into multiple image blocks according to the division ratio; the feature point pairs are processed by Random Sample Consensus to obtain the global image transformation matrix; the matching error between the feature point pairs in each image block and the global image transformation matrix is calculated; the error weight corresponding to the image block is determined according to the matching error, and the error weight is processed by weighted least squares to obtain the local image transformation matrix of the image block.

[0093] Among them, the division ratio refers to the data preset for dividing the reference image and the image to be registered. For example, the division ratio can be 16*16, that is, the length and width of the reference image and the image to be registered are evenly divided into 16 parts, resulting in 16*16 image blocks. The division ratio can also be 20*20, etc. Specifically, it can be custom-set according to the actual situation (such as the image size, the computing power of the electronic device, etc.). This example embodiment does not make special limitations on this.

[0094] The feature point pairs in the entire image can be processed by Random Sample Consensus to obtain the global image transformation matrix. After obtaining the global image transformation matrix, the matching error between the feature point pairs in each image block and the global image transformation matrix can be calculated. Specifically, the feature points in each image block can be transformed according to the global image transformation matrix to obtain the transformed pixel points. The Euclidean distance between the coordinates of the transformed pixel point and the real pixel point can be used as the matching error between the feature point pairs in each image block and the global image transformation matrix.

[0095] The matching error threshold can be obtained, and target feature point pairs with a matching error smaller than the matching error threshold are selected. Furthermore, the error weight of the image block corresponding to the target feature point pair can be determined according to the matching error of the target feature point pair. The significance of this error weight lies in measuring the matching degree between the target feature point pair and the global image transformation matrix. The higher the error weight, the better the matching. Finally, the error weight can be processed by weighted least squares to obtain the local image transformation matrix of the image block.

[0096] By calculating the local image transformation matrix and then remapping the pixel points in the image to be registered through the local image transformation matrix, it can effectively prevent the error between the image points in a certain image block and the global image transformation matrix from being too large, and improve the accuracy of the image texture alignment result.

[0097] Optionally, remapping the pixel points in the image to be registered according to the image transformation matrix can be achieved through the following steps: the local image transformation matrix corresponding to each image block in the image to be registered can be obtained; based on the local image transformation matrix of the image block where the current pixel point is located and the local image transformation matrix of the adjacent image blocks of the current pixel point, the current pixel point is remapped to obtain the target exposure image.

[0098] Figure 5 Schematically shows a schematic diagram of the principle of remapping through a local image transformation matrix in an exemplary embodiment of the present disclosure.

[0099] Refer to Figure 5 As shown, for the current pixel point A(x, y) in the fourth image block, the horizontal line passing through the current pixel point A intersects the left and right boundaries at points B and C. The length and width of the fourth image block are width and height respectively. The local image transformation matrix corresponding to the first image block is HG1, the local image transformation matrix corresponding to the second image block is HG2, the local image transformation matrix corresponding to the third image block is HG3, and the local image transformation matrix corresponding to the fourth image block is HG4. Point B is transformed through HG1 to obtain point B1, point B is transformed through HG3 to obtain point B3, point C is transformed through HG2 to obtain point C2, and point C is transformed through HG4 to obtain point C4. Then, the pixel point obtained after remapping the current pixel point A can be expressed as relation (3):

[0100]

[0101] wherein, A w can represent the pixel point obtained after remapping the current pixel point A, and W1, W2, W3, and W4 can respectively represent four weight values.

[0102] Remap the current pixel using the local image transformation matrix of the image block where the current pixel is located and the local image transformation matrices of neighboring image blocks. That is, the remapping of the pixel is jointly determined by at least four local image transformation matrices, which can effectively prevent the overlapping or separation of image blocks and pixels in the image blocks after remapping, and improve the image quality of the exposure image after image texture alignment.

[0103] In an exemplary embodiment, the image fusion of the original exposure image after registration alignment can be achieved through the steps in Figure 6 . Referring to Figure 6 shown, it may specifically include:

[0104] Step S610, perform image fusion on the first exposure image to generate a target fusion image;

[0105] Step S620, perform image fusion on the target fusion image and the second exposure image to generate a target output image.

[0106] Among them, the target fusion image refers to the image generated after performing image fusion on the first exposure images under the same exposure parameters. For example, for 3 frames of original exposure images collected at the EV0 level, these 3 frames of original exposure images at the EV0 level can be fused to obtain one frame of the original exposure image at the EV0 level, that is, the target fusion image.

[0107] Optionally, the image fusion of the first exposure image can be achieved through the steps in Figure 7 . Referring to Figure 7 shown, it may specifically include:

[0108] Step S710, use the exposure images in the first exposure image other than the reference image as the images to be fused;

[0109] Step S720, obtain a preset division ratio, and divide the reference image and the images to be fused into multiple image blocks according to the division ratio;

[0110] Step S730, calculate the image block difference between the corresponding image blocks in the reference image and the images to be fused;

[0111] Step S740, determine the image block type of the image block according to the image block difference;

[0112] Step S750, perform image fusion on the first exposure image based on the image block type to generate a target fusion image.

[0113] Among them, the division ratio refers to the data preset for dividing the reference image and the image to be fused. This division ratio can be the same as that for dividing the reference image and the image to be registered, and can divide the reference image and the image to be fused into multiple image blocks of the same size, facilitating image fusion. The division ratio in this embodiment may be the same as or different from the division ratio for dividing the reference image and the image to be registered. This exemplary embodiment does not make special limitations on this.

[0114] The image block difference refers to the pixel value difference between the corresponding image blocks in the reference image and the image to be fused. Furthermore, the image block type of the image block can be determined based on the image block difference. For example, the image block type can be a smooth image block, which can indicate that the image block difference between the reference image and the image to be registered is small, and the image content in the image block belongs to a static scene; the image block type can also be a moving image block, which can indicate that the image block difference between the reference image and the image to be registered is large, and the image content in the image block belongs to a dynamic scene. Of course, the image block type can also be other distinguishing methods. This exemplary embodiment does not make special limitations on this.

[0115] Different image block types can be determined to distinguish image blocks, and different fusion methods can be adopted for image blocks of different image block types, which can effectively improve the accuracy of the image fusion result of the first exposure image.

[0116] Optionally, if it is determined that the image block difference is less than or equal to the smooth threshold, the image block corresponding to the image block difference can be determined as a smooth image block, and the average value of the pixel points at the current position in the smooth image block of each first exposure image can be used as the fused pixel point at the current position, and the target fused image can be generated based on the fused pixel points.

[0117] Optionally, if it is determined that the image block difference is greater than the smooth threshold, the image block corresponding to the image block difference is determined as a moving image block, and the pixel points of the moving image block in the reference image can be directly used as the fused pixel points of the moving image block in the first exposure image, and the target fused image can be generated based on the fused pixel points.

[0118] Figure 8 Schematically shows a schematic diagram of the principle of image fusion based on image block types in an exemplary embodiment of the present disclosure.

[0119] Refer to Figure 8As shown, taking the image fusion of the reference image 810 and the image to be fused 820 as an example, the reference image 810 and the image to be fused 820 can be divided into multiple image blocks according to a preset division ratio, such as the division ratio can be 5*5. Then, the image block differences between the corresponding image blocks in the reference image 810 and the image to be fused 820 can be calculated. For example, calculate the image block difference between image block 1 in the reference image 810 and image block 1 in the image to be fused 820. If it is determined that the image block difference of image block 1 is less than or equal to the smoothing threshold, it can be determined that image block 1 corresponding to the image block difference is a smooth image block. Furthermore, the average value of the pixel points in image block 1 of the reference image 810 and image block 1 of the image to be fused 820 can be used as the fused pixel point at the current position to obtain the fused image block 831. If it is determined that the image block difference of image block 2 is greater than the smoothing threshold, it can be determined that image block 2 corresponding to the image block difference is a moving image block. Furthermore, the pixel points of image block 2 in the reference image 810 can be directly used as the fused pixel points of image block 2 to obtain the fused image block 832. Finally, the target fused image 830 can be generated based on the obtained fused image block 831 and the fused image block 832.

[0120] In an exemplary embodiment, the image fusion of the target fused image and the second exposure image can be achieved through the following steps: the brightness weight and the difference weight can be determined based on the target fused image and the second exposure image; the brightness mapping data is obtained, and the fusion weight is determined according to the brightness mapping data, the brightness weight, and the difference weight; the target fused image and the second exposure image are fused through the fusion weight to generate the target output image.

[0121] Among them, the brightness weight and the difference weight refer to the weight data determined based on the image data in the image to be fused. For example, the brightness weight can be determined according to the pixel data of the brighter image in the image to be fused, and the difference weight can be determined according to the pixel data difference between the images to be fused.

[0122] The fusion weight can be determined according to the brightness mapping data, the brightness weight, and the difference weight. Furthermore, the target fused image and the second exposure image are fused according to the fusion weight to generate the target output image. Through the fusion weight that combines the brightness mapping data, the brightness weight, and the difference weight, the texture clarity in the target output image obtained by fusion can be further improved, effectively improving the image quality of the target output image.

[0123] Optionally, the luminance weight and the difference weight can be determined based on the target fusion image and the second exposure image through the following steps: The one with a larger exposure amount among the target fusion image and the second exposure image can be used as the bright image, and the one with a smaller exposure amount among the target fusion image and the second exposure image can be used as the dark image; perform an inversion operation on the pixel values in the bright image, and map the bright image with the inverted pixel values through a mapping curve to obtain the luminance weight; determine the pixel difference between the bright image and the dark image, and map the pixel difference through a mapping curve to obtain the difference weight.

[0124] For example, for 1 frame of original exposure image collected at EV+2, 3 frames of original exposure images collected at EV0, and 1 frame of original exposure image collected at EV-2, the 3 frames of original exposure images (i.e., the first exposure images belonging to the same exposure parameter) collected at EV0 can be first subjected to image fusion to obtain the target fusion image at EV0. Then, the target fusion image at EV0 can be respectively fused with 1 frame of original exposure image (the second exposure image) at EV-2 and 1 frame of original exposure image (the second exposure image) at EV+2.

[0125] The target fusion image at EV0 can be first fused with 1 frame of original exposure image at EV+2, and then the fusion result of the two can be fused with 1 frame of original exposure image at EV-2.

[0126] Specifically, the target fusion image at EV0 can be first fused with 1 frame of original exposure image at EV+2:

[0127] The target fusion image at EV0 and the original exposure image at EV+2 can be first smoothed respectively. For example, the target fusion image at EV0 and the original exposure image at EV+2 can be smoothed through a 5*5 mean filter window. The filter window can be customarily set and is not specially limited here.

[0128] The one with a larger exposure amount among the smoothed target fusion image at EV0 and the original exposure image at EV+2 can be used as the bright image. In this step, the original exposure image at EV+2 can be used as the bright image, and the target fusion image at EV0 can be used as the dark image. An inversion operation can be performed on the pixel data of the bright image at EV+2, that is, (MAX - P EV+2 ), where MAX can represent the maximum range of the bit width of the exposure image. For example, if the bit width of the exposure image is 8 bit, then MAX can be 256; if the bit width of the exposure image is 10 bit, then MAX can be 1024; and then through Figure 9 the mapping curve 910 shown, (MAX - PEV+2 ) Map it to a preset bit width (such as 8-bit width) to obtain the luminance weight; then the pixel difference abs(P EV+2 -A*P EV0 ) between the bright image at EV+2 stops and the dark image at EV0 stop can be determined, and the pixel difference abs(P Figure 9 ) is mapped to the preset bit width (such as 8-bit width) through the mapping curve 910 shown in EV+2 -A*P EV0 ) to obtain the difference weight.

[0129] Then, image fusion can be performed on the EV0 stop fusion result obtained by fusion and one frame of the original exposure image at EV-2 stops:

[0130] First, smoothing processing can be performed on the EV0 stop fusion result and the original exposure image at EV-2 stops respectively. For example, the EV0 stop fusion result and the original exposure image at EV-2 stops can be smoothed through a 5*5 mean filtering window, and the filtering window can be customarily set and no special limitation is made here.

[0131] The one with a larger exposure amount in the smoothed EV0 stop fusion result and the original exposure image at EV-2 stops can be used as the bright image. In this step, the EV0 stop fusion result can be used as the bright image, and the original exposure image at EV-2 stops can be used as the dark image. The pixel data of the bright image at EV0 stop can be inverted, that is, (MAX-P EV0 ), where MAX can represent the maximum range of the bit width of the exposure image. For example, if the bit width of the exposure image is 8 bits, then MAX can be 256; if the bit width of the exposure image is 10 bits, then MAX can be 1024; then (MAX-P Figure 9 ) is mapped to the preset bit width (such as 8-bit width) through the mapping curve 910 shown in EV0 ) to obtain the luminance weight; then the pixel difference abs(P EV0 -B*P EV-2 ) between the bright image at EV0 stop and the dark image at EV-2 stop can be determined, and the pixel difference abs(P Figure 9 ) is mapped to the preset bit width (such as 8-bit width) through the mapping curve 910 shown in EV0 -B*P EV-2 ) to obtain the difference weight.

[0132] The calculation method of the general fusion weight can be expressed by the relational formula (4):

[0133] W mix =W Lum *W Diff / (255*255) (4)

[0134] Among them, W mix can represent the fusion weight, and W Lum can represent the brightness weight, and W Diff can represent the difference weight.

[0135] The process of image fusion that passes through can be represented by the relational expression (5):

[0136] P mix = P L *W mix + P S *Ratio*(1 - W mix ) (5)

[0137] Among them, P mix can represent the image fusion result, W mix can represent the fusion weight, P L can represent the pixel data corresponding to the bright image, P S can represent the pixel data corresponding to the dark image, Ratio can represent the brightness mapping data, that is, the brightness factor A and the brightness factor B, and Ratio can be determined according to the images to be fused. For example, for the image fusion of the target fusion image in the EV0 gear and the 1-frame original exposure image in the EV+2 gear, Ratio can take the brightness factor A; for the image fusion of the EV0 gear fusion result obtained by fusion and the 1-frame original exposure image in the EV-2 gear, Ratio can take the brightness factor B.

[0138] In an application scenario, the original exposure image can be the Raw domain image data collected by the camera module of the electronic device. Then, the registered and aligned original exposure images can be subjected to image fusion to generate a high-dynamic range image HDR. Then, the image signal processing pipeline (ISP Pipeline) can continue to perform processes such as brightening and tone mapping on the high-dynamic range image HDR to generate a target output image that can finally be output to and displayed on the display screen of the electronic device.

[0139] In summary, in this exemplary embodiment, the acquired original exposure image can be first subjected to registration and alignment processing to obtain the registered and aligned original exposure image, and then the registered and aligned original exposure image can be subjected to image fusion to generate the target output image. Performing registration and alignment on the original exposure image before image fusion can ensure the accuracy of the image content expression at the same position in the original exposure image, effectively reduce the ghosting phenomenon in the target output image obtained after image fusion, thereby enhancing the clarity of texture details and improving the image quality of the target output image.

[0140] It should be noted that the above-mentioned drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, rather than for limiting purposes. It is easy to understand that the processes shown in the above-mentioned drawings do not indicate or limit the chronological order of these processes. Additionally, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0141] Further, referring to Figure 10 as shown, in the embodiment of this example, an image fusion device 1000 is further provided, which may include an image acquisition module 1010, an image alignment module 1020, and an image fusion module 1030. Among them:

[0142] The image acquisition module 1010 is configured to acquire original exposure images, where the original exposure images include a first exposure image belonging to the same exposure parameter and a second exposure image belonging to different exposure parameters;

[0143] The image alignment module 1020 is configured to perform registration and alignment processing on the original exposure images to obtain the registered and aligned original exposure images;

[0144] The image fusion module 1030 is configured to perform image fusion on the registered and aligned original exposure images to generate a target output image.

[0145] In an exemplary embodiment, the original exposure images may include a first exposure image belonging to the same exposure parameter and a second exposure image belonging to different exposure parameters.

[0146] In an exemplary embodiment, the image alignment module 1020 may include:

[0147] A brightness alignment unit, configured to perform brightness alignment processing on the second exposure image to obtain the second exposure image after brightness alignment;

[0148] An image texture alignment unit, configured to perform image texture alignment on the first exposure image and the second exposure image after brightness alignment to obtain a target exposure image.

[0149] In an exemplary embodiment, the brightness alignment unit may be used to:

[0150] Statistically analyze the brightness information corresponding to the second exposure image;

[0151] Determine the brightness mapping data between the second exposure images according to the brightness information;

[0152] Based on the brightness mapping data, perform brightness alignment processing on the second exposure image to obtain the second exposure image after brightness alignment.

[0153] In an exemplary embodiment, the brightness alignment unit may be used to:

[0154] Obtain a preset number of pixel blocks, and divide the pixel blocks in the second exposure image according to the number of pixel blocks;

[0155] Determine the target pixel blocks in the divided second exposure image;

[0156] Calculate the brightness histogram of the target pixel blocks, and determine the brightness information corresponding to the second exposure image according to the brightness histogram.

[0157] In an exemplary embodiment, the brightness alignment unit can be used to:

[0158] Obtain a preset brightness interval, and screen target brightness data in the brightness histogram according to the brightness interval;

[0159] Determine the brightness information corresponding to the second exposure image based on the target brightness data.

[0160] In an exemplary embodiment, the image texture alignment unit can be used to:

[0161] Screen a reference image in the first exposure image or the second exposure image after brightness alignment, and use the exposure images other than the reference image in the first exposure image and the second exposure image after brightness alignment as the images to be registered;

[0162] Determine the feature point pairs between the reference image and the images to be registered, and determine the image transformation matrix according to the feature point pairs;

[0163] Remap the pixel points in the images to be registered according to the image transformation matrix to obtain the target exposure image.

[0164] In an exemplary embodiment, the image texture alignment unit can be used to:

[0165] Obtain a preset division ratio, and divide the reference image and the images to be registered into multiple image blocks according to the division ratio;

[0166] Perform random sample consensus processing on the feature point pairs to obtain a global image transformation matrix;

[0167] Calculate the matching error between the feature point pairs in each image block and the global image transformation matrix;

[0168] Determine the error weight corresponding to the image block according to the matching error, and perform weighted least squares processing on the error weight to obtain the local image transformation matrix of the image block.

[0169] In an exemplary embodiment, the image texture alignment unit can be used to:

[0170] Obtain the local image transformation matrix corresponding to each image block in the to-be-registered image;

[0171] Based on the local image transformation matrix of the image block where the current pixel is located and the local image transformation matrix of the neighboring image blocks of the current pixel, remap the current pixel to obtain the target exposure image.

[0172] In an exemplary embodiment, the image fusion module 1030 may include:

[0173] A first fusion unit, configured to perform image fusion on the first exposure image to generate a target fusion image;

[0174] A second fusion unit, configured to perform image fusion on the target fusion image and the second exposure image to generate a target output image.

[0175] In an exemplary embodiment, the first fusion unit may be configured to:

[0176] Use the exposure image in the first exposure image other than the reference image as the image to be fused;

[0177] Obtain a preset division ratio, and divide the reference image and the image to be fused into multiple image blocks according to the division ratio;

[0178] Calculate the image block difference between the corresponding image blocks in the reference image and the image to be fused;

[0179] Determine the image block type of the image block according to the image block difference;

[0180] Perform image fusion on the first exposure image based on the image block type to generate a target fusion image.

[0181] In an exemplary embodiment, the first fusion unit may be configured to:

[0182] If it is determined that the image block difference is less than or equal to the smoothing threshold, determine that the image block corresponding to the image block difference is a smooth image block;

[0183] Use the average value of the pixel points at the current position in the smooth image block of each first exposure image as the fusion pixel point at the current position;

[0184] Generate a target fusion image based on the fusion pixel points.

[0185] In an exemplary embodiment, the first fusion unit may be configured to:

[0186] If it is determined that the difference between the image blocks is greater than the smoothing threshold, it is determined that the image block corresponding to the difference between the image blocks is a moving image block;

[0187] Use the pixel points of the moving image block in the reference image as the fused pixel points of the moving image block in the first exposure image;

[0188] Generate a target fused image based on the fused pixel points.

[0189] In an exemplary embodiment, the second fusion unit may be used to:

[0190] Determine the luminance weight and the difference weight based on the target fused image and the second exposure image;

[0191] Obtain luminance mapping data, and determine the fusion weight according to the luminance mapping data, the luminance weight, and the difference weight;

[0192] Perform image fusion on the target fused image and the second exposure image through the fusion weight to generate a target output image.

[0193] In an exemplary embodiment, the second fusion unit may be used to:

[0194] Use the one with the larger exposure amount between the target fused image and the second exposure image as the bright image, and use the one with the smaller exposure amount between the target fused image and the second exposure image as the dark image;

[0195] Perform an inversion operation on the pixel values in the bright image, and map the bright image with the inverted pixel values through a mapping curve to obtain the luminance weight;

[0196] Determine the pixel difference between the bright image and the dark image, and map the pixel difference through the mapping curve to obtain the difference weight.

[0197] In an exemplary embodiment, the image fusion module 1030 may be used to:

[0198] Perform image fusion on the registered and aligned original exposure images to generate a high-dynamic range image;

[0199] Perform tone mapping on the high-dynamic range image to generate a target output image.

[0200] The specific details of each module in the above device have been described in detail in the implementation manner of the method part. The undisclosed detailed content can be referred to the implementation manner content of the method part, and thus will not be elaborated here.

[0201] Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method, or a program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuitry", "module", or "system" here.

[0202] Taking the mobile terminal 1100 in Figure 11 as an example, the structure of the electronic device will be described exemplarily. Those skilled in the art should understand that, except for the components specifically for mobile purposes, Figure 11 the structure in

[0203] can also be applied to fixed-type devices. Figure 11 As shown in Figure 11 , the mobile terminal 1100 may specifically include: a processor 1101, a memory 1102, a bus 1103, a mobile communication module 1104, an antenna 1, a wireless communication module 1105, an antenna 2, a display screen 1106, a camera module 1107, an audio module 1108, a power module 1109, and a sensor module 1110.

[0204] The processor 1101 may include one or more processing units. For example, the processor 1101 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor, and / or an NPU (Neural-Network Processing Unit), etc.

[0205] An encoder can encode (i.e., compress) an image or video to reduce the data size for easy storage or transmission. A decoder can decode (i.e., decompress) the encoded data of an image or video to restore the image or video data. The mobile terminal 1100 can support one or more encoders and decoders, such as image formats like JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap), and video formats like MPEG (Moving Picture Experts Group) 1, MPEG10, H.1063, H.1064, HEVC (High Efficiency Video Coding).

[0206] The processor 1101 can be connected to the memory 1102 or other components via the bus 1103.

[0207] The memory 1102 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 1101 executes various functional applications and data processing of the mobile terminal 1100 by running the instructions stored in the memory 1102. The memory 1102 can also store application data, such as storing files like images and videos.

[0208] The communication function of the mobile terminal 1100 can be implemented through the mobile communication module 1104, antenna 1, wireless communication module 1105, antenna 2, modulation and demodulation processor, and baseband processor, etc. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 1104 can provide 3G, 4G, 5G, etc. mobile communication solutions for the mobile terminal 1100. The wireless communication module 1105 can provide wireless communication solutions such as wireless local area network, Bluetooth, and near field communication for the mobile terminal 1100.

[0209] The display screen 1106 is used to implement the display function, such as displaying the user interface, images, videos, etc. The camera module 1107 is used to implement the shooting function, such as shooting images, videos, etc. The audio module 1108 is used to implement the audio function, such as playing audio, collecting voices, etc. The power module 1109 is used to implement the power management function, such as charging the battery, powering the device, monitoring the battery status, etc.

[0210] The sensor module 1110 can include one or more sensors for implementing corresponding sensing and detection functions.

[0211] Exemplary embodiments of the present disclosure also provide a computer-readable storage medium, on which a program product is stored that can implement the methods described above in this specification. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Methods" section above in this specification.

[0212] It should be noted that the computer-readable medium shown in the present disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium can include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0213] In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination of the above.

[0214] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0215] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present disclosure are pointed out by the claims.

[0216] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the figures, and various modifications and changes may be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image fusion method, characterized in that, Including: Obtaining an original exposure image, where the original exposure image includes a first exposure image belonging to the same exposure parameter and a second exposure image belonging to different exposure parameters; Performing registration and alignment processing on the original exposure image to obtain a target exposure image; Performing image fusion on the target exposure image to generate a target output image; The performing registration and alignment processing on the original exposure image to obtain a target exposure image includes: Performing brightness alignment processing on the second exposure image to obtain a second exposure image with brightness aligned; Performing image texture alignment on the first exposure image and the second exposure image with brightness aligned to obtain a target exposure image; Among them, the performing image texture alignment on the first exposure image and the second exposure image with brightness aligned to obtain a target exposure image includes: screening a reference image in the first exposure image or the second exposure image with brightness aligned, and using the exposure images other than the reference image in the first exposure image and the second exposure image with brightness aligned as images to be registered; determining feature point pairs between the reference image and the images to be registered, and determining an image transformation matrix according to the feature point pairs; remapping pixel points in the images to be registered according to the image transformation matrix to obtain a target exposure image; The remapping pixel points in the images to be registered according to the image transformation matrix to obtain a target exposure image includes: obtaining local image transformation matrices corresponding to each image block in the images to be registered; remapping the current pixel point based on the local image transformation matrix of the image block where the current pixel point is located and the local image transformation matrix of the adjacent image block of the current pixel point to obtain a target exposure image; The performing image fusion on the target exposure image to generate a target output image includes: performing image fusion on the first exposure image to generate a target fusion image; performing image fusion on the target fusion image and the second exposure image to generate a target output image; Among them, generating a target fusion image includes: using the exposure images other than the reference image in the first exposure image as images to be fused; obtaining a preset division ratio, and dividing the reference image and the images to be fused into multiple image blocks according to the division ratio; calculating the image block difference between the corresponding image blocks of the reference image and the images to be fused; determining the image block type of the image blocks according to the image block difference; performing image fusion on the first exposure image based on the image block type to generate a target fusion image.

2. The method according to claim 1, characterized in that The performing brightness alignment processing on the second exposure image to obtain a second exposure image with brightness aligned includes: Counting the brightness information corresponding to the second exposure image; Determining brightness mapping data between each of the second exposure images according to the brightness information; Performing brightness alignment processing on the second exposure image based on the brightness mapping data to obtain a second exposure image with brightness aligned.

3. The method according to claim 2, wherein The counting the brightness information corresponding to the second exposure image includes: Obtain a preset number of pixel blocks, and divide the pixel blocks in the second exposure image according to the number of pixel blocks; Determine the target pixel blocks in the second exposure image after division; Calculate the brightness histogram of the target pixel blocks, and determine the brightness information corresponding to the second exposure image according to the brightness histogram.

4. The method according to claim 3, characterized in that, The determining the brightness information corresponding to the second exposure image according to the brightness histogram includes: Obtain a preset brightness interval, and filter target brightness data in the brightness histogram according to the brightness interval; Determine the brightness information corresponding to the second exposure image based on the target brightness data.

5. The method according to claim 1, characterized in that The determining the image transformation matrix according to the feature point pairs includes: Obtain a preset division ratio, and divide the reference image and the image to be registered into multiple image blocks according to the division ratio; Perform random sample consensus processing on the feature point pairs to obtain a global image transformation matrix; Calculate the matching error between the feature point pairs in each image block and the global image transformation matrix; Determine the error weight corresponding to the image block according to the matching error, and perform weighted least squares processing on the error weight to obtain the local image transformation matrix of the image block.

6. The method according to claim 1, wherein The performing image fusion on the first exposure image based on the image block type to generate a target fusion image includes: If it is determined that the image block difference is less than or equal to the smoothing threshold, determine the image block corresponding to the image block difference as a smooth image block; Take the average value of the pixel points at the current position in the smooth image block of each first exposure image as the fusion pixel point at the current position; Generate a target fusion image based on the fusion pixel points.

7. The method according to claim 6, wherein The performing image fusion on the first exposure image based on the image block type to generate a target fusion image includes: If it is determined that the image block difference is greater than the smoothing threshold, determine the image block corresponding to the image block difference as a moving image block; Take the pixel points of the moving image block in the reference image as the fusion pixel points of the moving image block in the first exposure image; Generate a target fusion image based on the fusion pixel points.

8. The method according to claim 1, wherein The performing image fusion on the target fusion image and the second exposure image to generate a target output image includes: Determine the brightness weight and the difference weight based on the target fusion image and the second exposure image; Obtain brightness mapping data, and determine the fusion weight according to the brightness mapping data, the brightness weight, and the difference weight; Perform image fusion on the target fusion image and the second exposure image through the fusion weight to generate a target output image.

9. The method according to claim 8, characterized in that, The determining the brightness weight and the difference weight based on the target fusion image and the second exposure image includes: Take the one with the larger exposure amount in the target fusion image and the second exposure image as the bright image, and take the one with the smaller exposure amount in the target fusion image and the second exposure image as the dark image; Perform an inversion operation on the pixel values in the bright image, and map the bright image with the inverted pixel values through a mapping curve to obtain the brightness weight; Determine the pixel difference between the bright image and the dark image, and map the pixel difference through the mapping curve to obtain the difference weight.

10. The method according to claim 1, wherein The image fusion of the target exposure image to generate a target output image includes: Perform image fusion on the target exposure image to generate a high-dynamic-range image; Perform tone mapping on the high-dynamic-range image to generate a target output image.

11. An image fusion device, characterized in that, It includes: An image acquisition module for acquiring an original exposure image, where the original exposure image includes a first exposure image belonging to the same exposure parameter and a second exposure image belonging to different exposure parameters; An image alignment module for performing registration and alignment processing on the original exposure image to obtain a target exposure image; An image fusion module for performing image fusion on the target exposure image to generate a target output image; The registration and alignment processing of the original exposure image to obtain a target exposure image includes: performing brightness alignment processing on the second exposure image to obtain the second exposure image after brightness alignment; performing image texture alignment on the first exposure image and the second exposure image after brightness alignment to obtain a target exposure image; Among them, the performing image texture alignment on the first exposure image and the second exposure image after brightness alignment to obtain a target exposure image includes: screening a reference image in the first exposure image or the second exposure image after brightness alignment, and using the exposure images other than the reference image in the first exposure image and the second exposure image after brightness alignment as images to be registered; determining the feature point pairs between the reference image and the images to be registered, and determining an image transformation matrix according to the feature point pairs; remapping the pixel points in the images to be registered according to the image transformation matrix to obtain a target exposure image; The remapping the pixel points in the images to be registered according to the image transformation matrix to obtain a target exposure image includes: obtaining the local image transformation matrix corresponding to each image block in the images to be registered; based on the local image transformation matrix of the image block where the current pixel point is located and the local image transformation matrix of the adjacent image block of the current pixel point, remapping the current pixel point to obtain a target exposure image; The performing image fusion on the target exposure image to generate a target output image includes: performing image fusion on the first exposure image to generate a target fusion image; performing image fusion on the target fusion image and the second exposure image to generate a target output image; Among them, generating a target fusion image includes: using the exposure images other than the reference image in the first exposure image as images to be fused; obtaining a preset division ratio, and dividing the reference image and the images to be fused into multiple image blocks according to the division ratio; calculating the image block difference between the corresponding image blocks in the reference image and the images to be fused; determining the image block type of the image block according to the image block difference; performing image fusion on the first exposure image based on the image block type to generate a target fusion image.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 10.

13. An electronic device, characterized in that, Comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the method according to any one of claims 1 to 10 by executing the executable instructions.

Images