Multi-exposure image fusion system, method and electronic device

Abstract

This invention provides a multi-exposure image fusion system, method, and electronic device. The system includes: an image sequence alignment and correction module, comprising a selection submodule and an image sequence alignment and correction network. The selection submodule is configured to select a reference exposure level image, and the image sequence alignment and correction network is configured to align and correct other exposure level images based on the reference exposure level image; an image fusion deep network, configured to fuse the aligned and corrected multi-exposure image sequence to generate a multi-exposure fused image; and a visual quality-driven perception optimization module, configured to evaluate the multi-exposure fused image using a fused image visual quality evaluation model and optimize the image fusion deep network based on the evaluation results. This invention, by performing multi-exposure image fusion based on a deep network, can effectively capture image details and better understand the content of multi-exposure images, thereby achieving fast and efficient multi-exposure image fusion.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimedia digital image processing technology, and in particular to a multi-exposure image fusion system, method and electronic device. Background Technology

[0002] Multi-exposure image fusion is an important development direction in multimedia signal processing, providing core technologies for high dynamic range (HDR) imaging systems. In recent years, with the rapid development of computational imaging and the significant improvement in data transmission efficiency of mobile communication systems, people have increasingly higher demands for the visual quality and realism of captured images. Compared to ordinary images, HDR imaging technology can provide a wider dynamic range and richer image details, driving digital media towards higher quality and richer information, and has become a core competitive advantage for next-generation computational imaging and display devices. Currently, HDR imaging technology has been widely applied in many fields such as smart mobile devices, the gaming and entertainment industry, aerospace systems, and medical imaging equipment. For example, smartphones, tablets, and other mobile devices have begun to offer HDR shooting modes, bringing users a better photography experience; many video games are also being produced using HDR to provide a more realistic gaming experience and visual effects; with the help of HDR and multispectral technologies, satellite imaging systems have made it possible to reproduce spectral information invisible to the naked eye; at the same time, HDR and high sampling accuracy medical imaging systems have greatly improved the accuracy of patient diagnosis and the success rate of surgical procedures. As a core technology of high dynamic range imaging, multi-exposure image fusion aims to combine multiple images with different exposure levels into a single image that is clear in detail, rich in color, and full in content, making it approximate the real natural scene observed by the human eye. With the development of high dynamic range imaging and display technology, multi-exposure image fusion has gradually attracted widespread attention from academic and industrial communities both domestically and internationally.

[0003] With the continuous advancement of deep learning technology in recent years, neural network-based deep learning methods have gradually become the mainstream of research and application in the field of multi-exposure image fusion. At present, most multi-exposure image fusion algorithms can only achieve good results in relatively simple static scenes. However, for complex scenes, especially when moving objects are present, the fusion results will show obvious distortions such as halos and ghosting. Therefore, researchers generally design different visual quality evaluation models for fused images for static and dynamic scenes, and no unified fused image quality evaluation system has been formed. At the same time, research on perceptual optimization for multi-exposure image fusion has just begun. Currently, the applicant and research team have achieved preliminary results in perceptual optimization for static scenes, but there has been little progress in perceptual optimization for complex and dynamic scenes. Therefore, the following problems need to be solved in the quality evaluation and perceptual optimization of multi-exposure image fusion: (1) Factors affecting the visual quality of multi-exposure image fusion in complex scenes and their mechanisms of action; (2) Fusion mechanism of different visual features in the process of multi-exposure image fusion and its mathematical expression; (3) Perceptual optimization of multi-exposure image fusion based on visual quality evaluation models in complex scenes.

[0004] Therefore, it is necessary to propose an efficient and accurate multi-exposure image fusion system to generate high-quality multi-exposure fused images.

[0005] It should be noted that the information disclosed in the background section of this invention is intended only to enhance the understanding of the general background of this invention, and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a multi-exposure image fusion system, method, and electronic device. By performing multi-exposure image fusion based on deep networks, image details can be effectively captured, and the content of multi-exposure images can be better understood, thereby achieving fast and efficient multi-exposure image fusion.

[0007] To achieve the above objectives, the present invention provides a multi-exposure image fusion system, comprising:

[0008] An image sequence alignment and correction module includes a selection submodule and an image sequence alignment and correction network. The selection submodule is configured to use images that meet preset conditions in a multi-exposure image sequence as reference exposure level images based on image brightness features. The image sequence alignment and correction network is configured to align and correct other exposure level images in the multi-exposure image sequence other than the reference exposure level images based on the reference exposure level images, so as to compensate for motion information in the multi-exposure image sequence.

[0009] An image fusion deep network is configured to fuse the aligned and corrected multi-exposure image sequence to generate a multi-exposure fused image; and

[0010] The visual quality-driven perception optimization module is configured to evaluate the multi-exposure fused image using a fused image visual quality evaluation model, and optimize the image fusion deep network based on the evaluation results.

[0011] Optionally, the selection submodule is configured as follows:

[0012] Based on image brightness features, the sum of the number of underexposed pixels and the number of overexposed pixels in each image of the multi-exposure image sequence is calculated; and

[0013] The image with the smallest sum of underexposed and overexposed pixels is used as the reference exposure level image.

[0014] Optionally, the selection submodule is configured to determine the reference exposure level image according to the following formula:

[0015]

[0016] Among them, X r Indicates the reference exposure level image; X k This represents the k-th image in a multi-exposure image sequence; ccount(·) is an operation to count the total number of underexposed and overexposed pixels in the image, where k represents the index of the multi-exposure image sequence and K represents the total number of images in the multi-exposure image sequence.

[0017] Optionally, the image sequence alignment and correction network includes a localization module, a grid generator, and a sampler; the localization module is configured to predict the affine matrix parameters between other exposure level images and the reference exposure level image; the grid generator is configured to construct a corresponding sampling grid based on the affine matrix parameters between other exposure level images and the reference exposure level image; the sampler is configured to calculate the result of the other exposure level images after alignment and correction transformation based on the sampling grid, the other exposure level images, and the reference exposure level image.

[0018] Optionally, the image fusion deep network includes a pyramid decomposition module, an image fusion module, and a weighted summation module; the pyramid decomposition module is configured to decompose the aligned and corrected multi-exposure image sequence into a multi-scale image pyramid; the image fusion module is configured to predict the weight map sequence corresponding to the aligned and corrected multi-exposure image sequence based on the multi-scale image pyramid; the weighted summation module is configured to perform a weighted summation on each image in the aligned and corrected multi-exposure image sequence based on the weight map sequence to generate a multi-exposure fused image, using the following calculation formula:

[0019]

[0020] Where Y represents the multi-exposure fused image; W k X represents the weight map corresponding to the k-th image in the aligned and corrected multi-exposure image sequence; k For the k-th image in the aligned and corrected multi-exposure image sequence, ⊙ represents the weighted summation operation, K represents the total number of images in the aligned and corrected multi-exposure image sequence, and k represents the corresponding image index.

[0021] Optionally, the visual quality-driven perception optimization module is configured as follows:

[0022] The multi-exposure fused image and the aligned and corrected multi-exposure image sequence are simultaneously input into the fused image visual quality evaluation model to calculate the perceptual loss function value; and

[0023] Based on the perceptual loss function value, the image fusion deep network is optimized using the backpropagation algorithm.

[0024] Optionally, the image fusion deep network is further configured to generate an optimized multi-exposure fused image Y according to the following formula. opt :

[0025]

[0026] Constraint: 0 ≤ Y ≤ 255

[0027] Where Q represents the fused image visual quality evaluation model, {X k} represents the aligned and corrected multi-exposure image sequence, Y represents the multi-exposure fused image, and the constraint condition is that the pixel value of each pixel in the multi-exposure fused image Y is greater than or equal to 0 and less than or equal to 255.

[0028] To achieve the above objectives, the present invention also provides a multi-exposure image fusion method, which uses the multi-exposure image fusion system described in any of the above claims to generate a multi-exposure fused image based on a multi-exposure image sequence.

[0029] Optionally, the multi-exposure image fusion method provided by the present invention further includes:

[0030] The image fusion deep network is optimized at least once; and

[0031] The optimized image fusion deep network is used to re-fuse the aligned and corrected multi-exposure image sequence to generate an optimized multi-exposure fused image.

[0032] To achieve the above objectives, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the multi-exposure image fusion method described above.

[0033] Compared with the prior art, the multi-exposure image fusion system, method and electronic device provided by the present invention have the following beneficial effects:

[0034] The multi-exposure image fusion system provided by this invention aligns and corrects a multi-exposure image sequence comprising multiple images at different exposure levels to compensate for motion information in the sequence. This eliminates the influence of global motion on the fusion result, minimizing any differences caused by motion and effectively improving the overall quality and coherence of the fused image, thus providing data support for multi-exposure image fusion. Since the image fusion deep network in this invention is built based on the characteristics of multi-exposure image fusion, it can effectively handle the complexity of combining images captured at different exposure levels. This allows it to learn and optimize the fusion process according to the specific needs of multiple exposure scenarios, effectively capturing fine details and preserving important features of each exposure, thereby generating high-quality multi-exposure fused images efficiently and accurately. By using a pre-built fusion image visual quality evaluation model to evaluate the multi-exposure fused image and optimizing the image fusion deep network based on the evaluation results, the visual quality of the output fused image can be assessed and enhanced in real time. This prioritizes the preservation of necessary image details and improves the overall aesthetic quality of the fusion result, contributing to a more effective and efficient multi-exposure image fusion algorithm. In summary, the multi-exposure image fusion system provided by this invention can efficiently generate high-quality multi-exposure fused images, which can help promote the development of deep learning multi-exposure image fusion technology, and will also greatly promote the development of the field of multi-exposure image fusion and even the field of computer vision.

[0035] Since the multi-exposure image fusion method and electronic device provided by this invention belong to the same inventive concept as the multi-exposure image fusion system provided by this invention, the multi-exposure image fusion method and electronic device provided by this invention have at least all the beneficial effects of the multi-exposure image fusion system provided by this invention. For details, please refer to the relevant description above. Therefore, the beneficial effects of the multi-exposure image fusion method and electronic device provided by this invention will not be described in detail here. Attached Figure Description

[0036] Figure 1 A framework diagram of a multi-exposure image fusion system provided in one embodiment of the present invention;

[0037] Figure 2A framework diagram of an image sequence alignment and correction module provided in one embodiment of the present invention;

[0038] Figure 3 A framework diagram of an image fusion deep network provided in one embodiment of the present invention;

[0039] Figure 4 A flowchart of a multi-exposure image fusion method provided in one embodiment of the present invention;

[0040] Figure 5 A framework diagram of an electronic device provided according to an embodiment of the present invention.

[0041] The reference numerals in the attached figures are explained as follows:

[0042] Image sequence alignment and correction module - 100; Selection submodule - 110; Image sequence alignment and correction network - 120; Localization module - 121; Mesh generator - 122; Sampler - 123;

[0043] Image fusion deep network-200; pyramid decomposition module-210; image fusion module-220; weighted summation module-230;

[0044] Visual quality-driven perception optimization module-300;

[0045] Processor-410; Communication interface-420; Memory-430; Communication bus-440. Detailed Implementation

[0046] The multi-exposure image fusion system, method, and electronic device proposed in this invention are further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, used only to facilitate and clarify the explanation of the purpose of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only used to complement the content disclosed in the specification, for those skilled in the art to understand and read, and are not intended to limit the implementation conditions of this invention. Any modifications to the structure, changes in proportions, or adjustments to the size, provided that the effects and objectives achieved by this invention are the same or similar, should still fall within the scope of the technical content disclosed in this invention.

[0047] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The singular forms “a,” “an,” and “the” include plural objects. The term “or” is generally used to mean “and / or,” the term “several” is generally used to mean “at least one,” and the term “at least two” is generally used to mean “two or more.” Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0048] Furthermore, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0049] The core idea of ​​this invention is to provide a multi-exposure image fusion system, method and electronic device. By performing multi-exposure image fusion based on deep networks, image details can be effectively captured and the content of multi-exposure images can be better understood, thereby achieving fast and efficient multi-exposure image fusion.

[0050] It should be noted that the multi-exposure image fusion method provided by the present invention can be applied to the multi-exposure image fusion system provided by the present invention. The multi-exposure image fusion system can be configured on an electronic device, wherein the electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a mobile phone, a tablet computer, or other hardware device with various operating systems.

[0051] To achieve the above-mentioned goals, this invention provides a multi-exposure image fusion system. Please refer to [the relevant documentation]. Figure 1 and Figure 2 ,in, Figure 1 This is a framework diagram of a multi-exposure image fusion method provided in one embodiment of the present invention. Figure 2 This is a framework diagram of an image sequence alignment and correction module provided according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, the multi-exposure image fusion system provided by the present invention includes an image sequence alignment and correction module 100, an image fusion deep network 200, and a visual quality-driven perception optimization module 300. The image sequence alignment and correction module 100 includes a selection submodule 110 and an image sequence alignment and correction network 120. The selection submodule 110 is configured to use images in the multi-exposure image sequence that meet preset conditions as reference exposure level images based on image brightness features. The image sequence alignment and correction network 120 is configured to align and correct other exposure level images in the multi-exposure image sequence other than the reference exposure level image based on the reference exposure level image, so as to compensate for motion information in the multi-exposure image sequence. The image fusion deep network 200 is configured to fuse the aligned and corrected multi-exposure image sequence to generate a multi-exposure fused image. The visual quality-driven perception optimization module 300 is configured to evaluate the multi-exposure fused image using a fused image visual quality evaluation model and optimize the image fusion deep network 200 according to the evaluation results.

[0052] Therefore, the multi-exposure image fusion system provided by this invention aligns and corrects a multi-exposure image sequence comprising multiple images at different exposure levels to compensate for motion information in the sequence. This eliminates the influence of global motion in the multi-exposure image sequence on the fusion result, minimizing any differences caused by motion and effectively improving the overall quality and coherence of the fused image, thus providing data support for multi-exposure image fusion. Since the image fusion deep network 200 in this invention is built based on the characteristics of multi-exposure image fusion, it can effectively handle the complexity of combining images captured at different exposure levels. This allows it to learn and optimize the fusion process according to the specific needs of multiple exposure scenarios, effectively capturing fine details and retaining important features of each exposure, thereby generating high-quality multi-exposure fused images efficiently and accurately. By using a pre-built fusion image visual quality evaluation model to evaluate the multi-exposure fused image and optimizing the image fusion deep network 200 based on the evaluation results, the visual quality of the output fused image can be evaluated and enhanced in real time. This prioritizes the retention of necessary image details and improves the overall aesthetic quality of the fusion result, thus contributing to a more effective and efficient multi-exposure image fusion algorithm. In summary, the multi-exposure image fusion system provided by this invention can efficiently generate high-quality multi-exposure fused images, which can help promote the development of deep learning multi-exposure image fusion technology, and will also greatly promote the development of the field of multi-exposure image fusion and even the field of computer vision.

[0053] In some exemplary embodiments, the selection submodule 110 is configured as follows:

[0054] Based on image brightness features, the sum of the number of underexposed pixels and the number of overexposed pixels in each image of the multi-exposure image sequence is calculated; and

[0055] The image with the smallest sum of underexposed and overexposed pixels is used as the reference exposure level image.

[0056] Therefore, by selecting the image with the smallest sum of underexposed and overexposed pixels as the reference exposure level image, a good foundation can be laid for subsequent accurate alignment of multi-exposure image sequences, thereby improving the overall quality and coherence of the fused image.

[0057] Specifically, the reference exposure level image can be obtained according to the following formula:

[0058]

[0059] Among them, X r Indicates the reference exposure level image; X kThis represents the k-th image in a multi-exposure image sequence; count(·) is an operation to count the total number of underexposed and overexposed pixels in the image, where k represents the index of the multi-exposure image sequence and K represents the total number of images in the multi-exposure image sequence.

[0060] After selecting a reference exposure image, a multi-source bilinear objective function is established between the multi-exposure image sequence and the reference exposure image to achieve dynamic region detection and conversion compensation to static regions in the multi-exposure image. The multi-source bilinear objective function aims to calculate the integrity and consistency between other exposure images and the reference exposure image. While keeping the static regions unchanged, it updates the dynamic regions to ensure their motion information is consistent with the reference image. The specific calculation formula for the multi-source bilinear objective function is shown below:

[0061]

[0062] Among them, g k (·) represents the brightness mapping function, which can map an image of any exposure level to the k-th exposure level. MBDS(·) is a multi-source bilinear objective function, E MBDS A multi-source bilinear objective function representing a multi-exposure image sequence and a reference exposure level image;

[0063] The expression for the multi-source bilinear objective function is shown below:

[0064]

[0065] Where, x' k X' represents the k-th exposure level image after brightness mapping. k Image patch in; x' r This represents the reference exposure level image X' after brightness mapping. r Image patch in; w k (·) is a weighting function designed to give greater weight to areas with normal exposure, X' k X' represents the exposure level image after brightness mapping. r Let x' represent the reference exposure level image after luminance mapping, and let x' represent the reference exposure level image X' after luminance mapping. r A specific image patch within the image.

[0066] Please continue to refer to this. Figure 2 ,like Figure 2As shown, in some exemplary embodiments, the image sequence alignment and correction network 120 includes a localization module 121, a grid generator 122, and a sampler 123; the localization module 121 is configured to predict the affine matrix parameters between other exposure level images and the reference exposure level image; the grid generator 122 is configured to construct a corresponding sampling grid based on the affine matrix parameters between other exposure level images and the reference exposure level image; the sampler 123 is configured to calculate the result of the other exposure level images after alignment and correction transformation based on the sampling grid, the other exposure level images, and the reference exposure level image.

[0067] Specifically, the localization module 121 aims to predict the affine matrix parameters θ of other exposure level images and a reference exposure level image. The grid generator 122 constructs a sampling grid based on the predicted transformation parameters, which is the output obtained by sampling and transforming a set of pixels in the input image. Assume the coordinates of a pixel in another exposure level image are... The pixel coordinates corresponding to the reference exposure level image are Spatial transformation function τ θ Let (G) be a two-dimensional affine transformation function, then, and The correspondence can be expressed as:

[0068]

[0069] Where, θ 11 ,θ 12 ,θ 13 ,θ 21 ,θ 22 and θ 23 τ represents the affine transformation parameter. θ (.) denotes the space transformation function, G i This represents a vector consisting of the coordinates of a single pixel in the reference exposure image. This represents the coordinates of a pixel in an image at any exposure level. This indicates the pixel coordinates corresponding to the reference exposure level image.

[0070] Finally, sampler 123 uses both the sampling grid and the input image as input, and calculates the results of other exposure level images after alignment correction transformation based on the multi-source bilinear objective function of the multi-exposure image sequence and the reference exposure level image mentioned above.

[0071] Please continue to refer to this. Figure 3 This is a framework diagram of an image fusion deep network 200 provided in one embodiment of the present invention. Figure 3As shown, in some exemplary embodiments, the image fusion deep network 200 includes a pyramid decomposition module 210, an image fusion module 220, and a weighted summation module 230; the pyramid decomposition module 210 is configured to decompose the aligned and corrected multi-exposure image sequence into a multi-scale image pyramid; the image fusion module 220 is configured to predict the weight map sequence corresponding to the aligned and corrected multi-exposure image sequence based on the multi-scale image pyramid; the weighted summation module 230 is configured to perform a weighted summation on each image in the aligned and corrected multi-exposure image sequence based on the weight map sequence to generate a multi-exposure fused image.

[0072] Therefore, by decomposing the aligned and corrected multi-exposure image sequence into a multi-scale image pyramid, multi-scale operations can be achieved for multi-exposure image fusion, effectively suppressing halo distortion in the fused image. A fast and efficient context aggregation network (image fusion module 220) is designed specifically for the multi-exposure image fusion task to predict the weight map of the multi-exposure image sequence (i.e., predict the weight map sequence corresponding to the aligned and corrected multi-exposure image sequence), which improves both the fusion quality and efficiency of the multi-exposure images. The final fused image result is obtained by weighted summing the predicted weight map sequence with the aligned and corrected multi-exposure image sequence.

[0073] Specifically, a multi-exposure fused image can be obtained using the following formula:

[0074]

[0075] Where Y represents the multi-exposure fused image; W k X represents the weight map corresponding to the k-th image in the aligned and corrected multi-exposure image sequence; k This represents the k-th image in the aligned and corrected multi-exposure image sequence, ⊙ indicates the weighted summation operation, K represents the total number of images in the aligned and corrected multi-exposure image sequence, and k represents the corresponding image index.

[0076] In some exemplary embodiments, the visual quality-driven perception optimization module 300 is configured as follows:

[0077] The multi-exposure fused image and the aligned and corrected multi-exposure image sequence are simultaneously input into the fused image visual quality evaluation model to calculate the perceptual loss function value; and

[0078] Based on the perceptual loss function value, the image fusion deep network 200 is optimized using the backpropagation algorithm.

[0079] Therefore, by designing a perceptual loss function based on the visual quality evaluation model of fused images, a real-time and efficient multi-exposure image fusion algorithm can be constructed. This algorithm combines the multi-exposure fused image (Y) generated by the image fusion deep network 200 with the aligned and corrected multi-exposure image sequence (X). k As input to the constructed fused image visual quality evaluation model, the model loss (i.e., the perceptual loss function value) is calculated, and the image fusion deep network 200 is optimized end-to-end through backpropagation. This can improve the fusion efficiency of multi-exposure images while ensuring the quality of the fused images.

[0080] In some exemplary embodiments, the image fusion deep network 200 is further configured to generate an optimized multi-exposure fused image Y according to the following formula. opt :

[0081]

[0082] Constraint: 0 ≤ Y ≤ 255

[0083] Where Q represents the fused image visual quality evaluation model, {X k} represents the aligned and corrected multi-exposure image sequence, and Y represents the multi-exposure fused image.

[0084] It should be noted that, as those skilled in the art will understand, 0≤Y≤255 means that the pixel value of each pixel in the multi-exposure fused image (Y) is greater than or equal to 0 and less than or equal to 255.

[0085] Furthermore, the multi-exposure fused image and the multi-exposure image sequence before alignment and correction (i.e., the original multi-exposure image sequence) can be simultaneously input into the fused image visual quality evaluation model to calculate the perceptual loss function value. Based on the perceptual loss function value, the image sequence alignment and correction network 120 and the image fusion depth network 200 are jointly optimized to construct a multi-exposure image fusion perceptual optimization model (a real-time and efficient multi-exposure image fusion algorithm).

[0086] Based on the same inventive concept, this invention also provides a multi-exposure image fusion method. This method employs the multi-exposure image fusion system described above, generating a multi-exposure fused image based on a multi-exposure image sequence. Since the multi-exposure image fusion method and the multi-exposure image fusion system provided by this invention belong to the same inventive concept, the multi-exposure image fusion method provided by this invention possesses at least all the beneficial effects of the multi-exposure image fusion system provided by this invention. For details, please refer to the relevant descriptions above; therefore, the beneficial effects of the multi-exposure image fusion method provided by this invention will not be elaborated upon here.

[0087] In some exemplary embodiments, the multi-exposure image fusion method provided by the present invention further includes:

[0088] The image fusion deep network 200 is optimized at least once; and

[0089] The optimized image fusion deep network 200 is used to re-fuse the aligned and corrected multi-exposure image sequence to generate an optimized multi-exposure fused image.

[0090] Therefore, by using the optimized image fusion depth network 200 to re-fuse the aligned and corrected multi-exposure image sequence, the quality of the final fused image can be effectively guaranteed.

[0091] Please refer to Figure 4 This is a flowchart of a multi-exposure image fusion method provided by an embodiment of the present invention. Figure 4 As shown, the multi-exposure image fusion method provided by the present invention includes the following steps:

[0092] Step S1: Compensate for motion information in the image sequence (multi-exposure image sequence) through image sequence alignment and correction;

[0093] Step S2: Based on the characteristics of multi-exposure image fusion, design an image fusion deep network 200;

[0094] Step S3: Based on the fused image visual quality evaluation model, design a perceptual loss function to optimize the image fusion deep network 200.

[0095] Specifically, in step S1, the method for aligning and correcting the image sequence includes the following steps:

[0096] S101. Calculate the reference exposure level image of the multi-exposure image sequence based on the brightness characteristics of the design, that is, select an image with an exposure level (meeting the preset conditions) as the reference image based on the image brightness characteristics.

[0097] S102. Construct an image sequence alignment and correction network 120 with a localization module 121, a mesh generator 122, and a sampler 123. The localization module 121 aims to predict the affine matrix parameters θ of other exposure level images and a reference exposure level image. The mesh generator 122 constructs a sampling mesh based on the predicted transformation parameters, which is the output obtained after sampling transformation of pixels in a set of input images. Finally, the sampler 123 uses both the sampling mesh and the input image as input to calculate the result of the alignment and correction transformation of other exposure level images.

[0098] In step S2, based on the characteristics of multi-exposure image fusion, an image fusion deep network 200 is designed, and the specific steps include:

[0099] S201. Decompose the multi-exposure image sequence pyramid into a multi-scale image pyramid to achieve multi-scale operation of multi-exposure image fusion.

[0100] S202. For the multi-exposure image fusion task, a fast and efficient context aggregation network is designed to predict the weight map of the multi-exposure image sequence.

[0101] S203. The weight map predicted by the image fusion deep network 200 is weighted and summed with the multi-exposure image sequence to obtain the final fused image result.

[0102] Based on the same inventive concept, the present invention also provides an electronic device, please refer to... Figure 5 This is a framework diagram of an electronic device provided in one embodiment of the present invention. Figure 5 As shown, the electronic device includes a processor 410 and a memory 430. The memory 430 stores a computer program. When the computer program is executed by the processor 410, it implements the multi-exposure image fusion method described above. Since the electronic device provided by this invention and the multi-exposure image fusion method provided by this invention belong to the same inventive concept, the electronic device provided by this invention has at least all the beneficial effects of the multi-exposure image fusion method provided by this invention. Therefore, the beneficial effects of the electronic device provided by this invention can be referred to the relevant descriptions of the beneficial effects of the multi-exposure image fusion method provided by this invention above, and will not be repeated here.

[0103] Please continue to refer to this. Figure 5 ,like Figure 5 As shown, the electronic device also includes a communication interface 420 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The communication bus 440 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 440 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 420 is used for communication between the aforementioned electronic device and other devices.

[0104] It should be noted that the processor 410 referred to in this invention can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 410 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and lines.

[0105] It should also be noted that the memory 430 can be used to store the computer program, and the processor 410 implements various functions of the electronic device by running or executing the computer program stored in the memory 430 and calling the data stored in the memory 430. The memory 430 may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable memory (PROM), electrically programmable memory (EPROM), electrically erasable programmable memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, random access memory is available in a variety of forms, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous random access memory (SDRAM), dual data rate synchronous random access memory (DDRSDRAM), enhanced synchronous random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), memory bus direct random access memory (RDRAM), direct memory bus dynamic random access memory (DRDRAM), and memory bus dynamic random access memory (RDRAM), etc.

[0106] In summary, compared with the prior art, the multi-exposure image fusion system, method, and electronic device provided by the present invention have the following beneficial effects:

[0107] 1. By designing an effective image sequence alignment and correction module, the influence of global motion on the fusion results of multi-exposure images can be eliminated, ensuring that any differences caused by motion are minimized, effectively improving the overall quality and coherence of the fused images, and providing data support for the fusion of multi-exposure images.

[0108] 2. Based on the multi-scale theory of visual perception system, a deep image fusion network 200 is designed. The aligned and corrected multi-exposure image sequence is decomposed into image pyramids and used as input to the deep neural network to predict the image pixel weight kernel. Finally, a fused image based on adaptive convolutional weight kernel is generated. This can effectively handle the complexity of combining images captured at different exposure levels. It can learn and optimize the fusion process according to the specific needs of multi-exposure scenes, effectively capture fine details and retain the important features of each exposure, so as to generate high-quality multi-exposure fused images efficiently and accurately.

[0109] 3. Based on the visual quality evaluation model, a low-complexity perceptual loss function is designed to jointly optimize the image sequence alignment and correction network 120 and the image fusion deep network 200, thereby constructing a multi-exposure image fusion perceptual optimization model. This model can evaluate and enhance the visual quality of the output fused image in real time, prioritize the preservation of necessary image details and improve the overall aesthetic quality of the fusion result, thus helping to achieve a more effective and efficient multi-exposure image fusion algorithm.

[0110] 4. Through in-depth research and modeling, we are committed to designing a visual quality-driven multi-exposure image fusion method. By designing a perceptual loss function and constructing a deep image fusion network 200, we contribute to advancing the development of deep learning-based multi-exposure image fusion.

[0111] 5. The efficient generation of high-quality multi-exposure fused images and related perceptual optimization will greatly promote the development of the field of multi-exposure image fusion and even the field of computer vision.

[0112] It should be noted that computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] It should be noted that the apparatus and methods disclosed in the embodiments herein can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments herein. In this regard, each block in a flowchart or block diagram may represent a module, program, or part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system to perform the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of this article can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0114] It should also be noted that the above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the protection scope of the present invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A multi-exposure image fusion system, characterized in that, include: An image sequence alignment and correction module includes a selection submodule and an image sequence alignment and correction network. The selection submodule is configured to use images that meet preset conditions in a multi-exposure image sequence as reference exposure level images based on image brightness features. The image sequence alignment and correction network is configured to align and correct other exposure level images in the multi-exposure image sequence other than the reference exposure level images based on the reference exposure level images, so as to compensate for motion information in the multi-exposure image sequence. An image fusion deep network is configured to fuse the aligned and corrected multi-exposure image sequence to generate a multi-exposure fused image; as well as The visual quality-driven perception optimization module is configured to evaluate the multi-exposure fused image using a fused image visual quality evaluation model, and optimize the image fusion deep network based on the evaluation results.

2. The multi-exposure image fusion system according to claim 1, characterized in that, The selection submodule is configured as follows: Based on image brightness features, the total number of underexposed pixels and overexposed pixels in each image of the multi-exposure image sequence is counted. as well as The image with the smallest sum of underexposed and overexposed pixels is used as the reference exposure level image.

3. The multi-exposure image fusion system according to claim 2, characterized in that, The selection submodule is configured to determine the reference exposure level image according to the following formula: Among them, X r Indicates the reference exposure level image; X k This represents the k-th image in a multi-exposure image sequence; count(·) is an operation to count the total number of underexposed and overexposed pixels in the image, where k represents the index of the multi-exposure image sequence and K represents the total number of images in the multi-exposure image sequence.

4. The multi-exposure image fusion system according to claim 1, characterized in that, The image sequence alignment and correction network includes a localization module, a grid generator, and a sampler; the localization module is configured to predict the affine matrix parameters between other exposure level images and the reference exposure level image; the grid generator is configured to construct a corresponding sampling grid based on the affine matrix parameters between other exposure level images and the reference exposure level image; The sampler is configured to calculate, based on the sampling grid, other exposure level images, and the reference exposure level image, the result of alignment and correction transformation of other exposure level images.

5. The multi-exposure image fusion system according to claim 1, characterized in that, The image fusion deep network includes a pyramid decomposition module, an image fusion module, and a weighted summation module. The pyramid decomposition module is configured to decompose the aligned and corrected multi-exposure image sequence into a multi-scale image pyramid. The image fusion module is configured to predict a weight map sequence corresponding to the aligned and corrected multi-exposure image sequence based on the multi-scale image pyramid. The weighted summation module is configured to perform a weighted summation on each image in the aligned and corrected multi-exposure image sequence based on the weight map sequence to generate a multi-exposure fused image. The calculation formula is as follows: Where Y represents the multi-exposure fused image; W k X represents the weight map corresponding to the k-th image in the aligned and corrected multi-exposure image sequence; k Let represent the k-th image in the aligned and corrected multi-exposure image sequence, ⊙ denotes the weighted summation operation, K represents the total number of images in the aligned and corrected multi-exposure image sequence, and k represents the corresponding image index.

6. The multi-exposure image fusion system according to claim 1, characterized in that, The visual quality-driven perception optimization module is configured as follows: The multi-exposure fused image and the aligned and corrected multi-exposure image sequence are simultaneously input into the fused image visual quality evaluation model to calculate the perceptual loss function value; as well as Based on the perceptual loss function value, the image fusion deep network is optimized using the backpropagation algorithm.

7. The multi-exposure image fusion system according to claim 6, characterized in that, The image fusion deep network is also configured to generate an optimized multi-exposure fusion image Y according to the following formula. opt : Constraint: 0 ≤ Y ≤ 255 Where Q represents the fused image visual quality evaluation model, {X k } represents the aligned and corrected multi-exposure image sequence, Y represents the multi-exposure fused image, and the constraint condition is that the pixel value of each pixel in the multi-exposure fused image Y is greater than or equal to 0 and less than or equal to 255.

8. A multi-exposure image fusion method, employing the multi-exposure image fusion system as described in any one of claims 1 to 7, to generate a multi-exposure fused image based on a multi-exposure image sequence.

9. The multi-exposure image fusion method according to claim 8, characterized in that, The method further includes: The image fusion deep network is optimized at least once; and The optimized image fusion deep network is used to re-fuse the aligned and corrected multi-exposure image sequence to generate an optimized multi-exposure fused image.

10. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the multi-exposure image fusion method as described in claim 8 or 9.

Images