Multi-exposure image integrated repairing and enhancing method and system based on composite optimization network

Abstract

The invention discloses a multi-exposure image integrated repairing and enhancing method and system based on a composite optimization network, and the method comprises the steps of image input, optical flow alignment, high-frequency enhancement, multi-branch fusion, output and the like. In high-frequency areas such as flame edges and face contours, detail recovery is obviously superior to that of a traditional method; and through targeted design, the model reasoning efficiency is improved, the reasoning time is shorter than that of similar models, and then the real-time performance of model reasoning is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of image processing and artificial intelligence network technology, and in particular to a multi-exposure image integrated restoration and enhancement method and system based on a composite optimization network. Background Art

[0002] Traditional image restoration methods based on image processing and machine learning (such as denoising, deblurring, and HDR reconstruction) rely on a single input image. This lack of information limits restoration effectiveness, especially in dynamic scenes where ghosting and detail loss are common. Existing deep learning-based methods (such as TMRNet) use multi-frame fusion, but as the network depth increases, early input image information is easily forgotten, resulting in insufficient restoration of high-frequency textures (such as edges and flame outlines). Furthermore, existing models have a large number of parameters and long inference times, making them difficult to meet real-time requirements. Summary of the Invention

[0003] In view of this, one of the objectives of the present invention is to provide an integrated multi-exposure image restoration and enhancement method based on a composite optimization network, which achieves high-quality image restoration and enhancement through joint processing of multi-exposure images, frequency separation enhancement and efficient feature fusion, while significantly reducing computing resource consumption.

[0004] One of the objectives of the present invention is achieved through the following technical solutions:

[0005] The multi-exposure image integrated restoration and enhancement method based on a composite optimization network comprises the following steps:

[0006] Step S1: Image input: multiple images with different exposure times are input. The images are taken in the same scene and arranged from low to high exposure time to ensure that the bright and dark details of the scene are covered. After normalization and gamma correction are performed on each image, they are stitched with the corresponding original image.

[0007] Step S2: Optical flow alignment: Use a pre-trained network to calculate the optical flow between multiple exposure images and estimate the motion vector of each image relative to the reference frame. Based on the optical flow results, each image is spatially transformed to ensure that all images are spatially aligned and eliminate misalignment caused by camera shake or object motion. The aligned images are then used to extract shallow features through convolution blocks, providing high-quality input for subsequent high-frequency enhancement.

[0008] Step S3: High-frequency enhancement: Through the average pooling and maximum pooling layers, the input feature map is separated into high-frequency information and low-frequency information, quickly capturing the high-frequency and low-frequency features of the image;

[0009] Step S4: Multi-branch fusion: A multi-branch convolution fusion structure is used to process high-frequency and low-frequency information separately through multi-path convolution combination;

[0010] Step S5: Output: The results of multiple high-frequency enhancement sub-networks are aligned with the reference frame through a simple convolution block and then fused to generate the final HDR image.

[0011] Furthermore, in step S1, it is assumed that there are i images, namely {R1, R2, R3, R4, ..., R i}, according to the exposure value of the image arranged from low to high, R1 is used as a reference, and each image is normalized. The formula is as follows:

[0012]

[0013] where Δt i Represents the exposure time of the i-th image, and then performs gamma correction. The formula is as follows:

[0014]

[0015] Where γ represents the gamma correction parameter. Next, the image data before and after gamma correction are spliced to obtain {I1, I2, I3, I4, ..., I i}, the formula is as follows:

[0016]

[0017] Furthermore, in step S2, {I1, I2, I3, I4, ..., I i} is input into the optical flow alignment model, making full use of the information of each image, and finally outputting the aligned fusion feature F align , the formula is as follows:

[0018] F align =f(I1,I2,I3,I4,……,I i ;θ) (4)

[0019] The function f(·) represents the optical flow alignment network model, and θ represents the parameters of this network model. After alignment, the shallow features are extracted and calculated as follows:

[0020] F=Conv 1×1 (ReLU(Conv 3×3 (F align )),ReLU(Conv 5×5 (F align )));

[0021] Among them, ReLU is a nonlinear activation function, Conv 1×1 、Conv 3×3 and Conv 5×5They represent convolution operations with kernel sizes of 1, 3, and 5, respectively. Using the convolution operation results of different kernel sizes, we can obtain information of different scales in shallow features, and then realize feature fusion through unit convolution, which can be input into the subsequent network structure for further feature extraction and learning.

[0022] Furthermore, in step S3, the shallow features F∈R extracted in step 2 are B×H×W×C , the feature separation of high and low frequency information is achieved through the following formula:

[0023] F L =Pooling(F) (5)

[0024] F H =F-Upsample(F L ) (6)

[0025] Among them, Pooling is a downsampling operation, and Upsample is a bilinear interpolation upsampling operation. The low-frequency features are obtained through pooling operation, F H ∈R B×H×W×C It is the high-frequency feature obtained by subtracting the low-frequency feature information from the original feature.

[0026] Furthermore, in step S4, after obtaining the clearly extracted high-frequency features F H Afterwards, the self-attention mechanism is used to globally enhance it to make full use of high-frequency information and obtain F H1 :

[0027] Q H =Conv 1×1 (F H ),K H =Conv 1×1 (F H ),V H =Conv 1×1 (F H )

[0028]

[0029] in is the scaling factor; d is the query vector dimension, which is used to stabilize the attention calculation and prevent the Softmax calculation gradient from disappearing. H1 With F L Based on this, a multi-branch structure is adopted to fully integrate high and low frequency features:

[0030] H=MBB(F H1 ) (7)

[0031] L=MBB(MBB(MBB(F L ))) (8)

[0032] In the above formula, H and L represent the high-frequency features and low-frequency features after dual-path convolution processing, respectively. MBB represents a dual-branch convolution structure, where branch 1 is used to enhance high-frequency details and branch 2 is used to preserve contours. The specific formula is as follows:

[0033] Y1=Conv 3×3 (X)

[0034] Y2=Conv 3×3 (Conv 3×3 (Conv 3×3 (X)))

[0035] Where Y1 and Y2 represent the output of branch 1 and branch 2 respectively, Conv 3×3 Represents a convolution operation with a kernel size of 3.

[0036] Furthermore, based on the dual branches, an additional dynamic deformable convolution cross branch is designed, namely branch 3. Branch 3 includes two parts: generating dynamic convolution kernel offset and generating deformable convolution output, as shown in the following formula:

[0037] ΔW d =σ(Conv 1×1 (H; L));

[0038] Where ΔW d Represents the generated dynamic convolution kernel offset, σ represents the sigmoid function, which maps the input to between 0 and 1. That is, through unit convolution, the high-frequency feature H and the low-frequency feature L are spliced and fused, and then sigmoid is used for constraint to achieve the ability to adaptively adjust the sampling position and solve motion blur and edge distortion. Based on the dynamic convolution kernel offset, the deformable convolution output is generated as shown in the following formula:

[0039]

[0040] Where K is the number of convolution kernels, and p is the feature map coordinate, which corresponds to the sampling point of the convolution kernel. It retains the conventional convolution structure while introducing deformation capability, thereby making the convolution operation have a larger receptive field, expanding the range of local features, and having stronger global modeling capabilities.

[0041] Furthermore, a frequency-aware cross-attention mechanism is set up to further enhance high-frequency features and compensate for low-frequency features. The specific calculation formula is as follows:

[0042] Q H =Conv 1×1 (H),KL =Conv 1×1 (L),V L =Conv 1×1 (L);

[0043]

[0044] in is a scaling factor; d is the query vector dimension, which is used to stabilize the attention calculation and prevent the gradient of the Softmax calculation from disappearing. The frequency-aware cross-attention mechanism uses high-frequency features to generate query vectors, and uses low-frequency features to generate key vectors and value vectors, and then performs attention calculation. Finally, feature fusion is performed to achieve mutual compensation and learning of high- and low-frequency features.

[0045] Furthermore, a multi-scale residual gated fusion mechanism is used to fuse the obtained high-frequency features and low-frequency features. The fusion mechanism includes branches I, II, and III, which correspond to high-frequency features, low-frequency features, and cross-frequency interactions, respectively. The three branches are weighted and fused to output the final high- and low-frequency fusion features. Branch I is used to perform dilated convolution on high-frequency features to enhance the receptive field of high-frequency features. The calculation formula is as follows:

[0046]

[0047] Where r represents the dilation ratio of the dilated convolution, and values of 1, 2, and 3 correspond to standard convolution, medium receptive field, and large receptive field, respectively, improving the network learning ability from local features to global features.

[0048] The branch II is used to optimize the low-frequency features and adopts the unit convolution and spatial attention mechanism to improve the performance of low-frequency features. The calculation formula is as follows:

[0049] F spatial =SA(Conv 1×1 (L));

[0050] Where SA represents the calculation of the spatial attention mechanism, F spatial It is the optimized low-frequency spatial domain feature;

[0051] The branch III is used for cross-frequency interactive fusion of low-frequency features, which is implemented using a weighted gating mechanism. The calculation formula is as follows:

[0052] G=σ(Conv 1×1 (H+L));

[0053] F gate =G·H+(1-G)·L;

[0054] Where G represents the gate weight, which is compressed by the sigmoid function and weighted by the fusion of high and low frequency features in the form of weights, and finally the fused feature F is obtained. gate ;

[0055] After completing the calculation of the above three branches, the above three branches are weighted fused and finally output. The calculation is as follows:

[0056] F final =α·F gate +β·F dilated +γ·F spatial ;

[0057] Among them F final It is the final output of the entire step 4, α+β+γ=1 and is a learnable parameter.

[0058] A second object of the present invention is to provide a multi-exposure image integrated restoration and enhancement system based on a composite optimization network, the system comprising:

[0059] An image input and processing module, which is used to receive external images. The images are taken in the same scene with exposure times arranged from low to high to ensure coverage of both bright and dark details of the scene. After normalization and gamma correction of each image, they are stitched with the corresponding original image;

[0060] The optical flow alignment module uses a pre-trained network to calculate the optical flow between multiple exposure images, estimates the motion vector of each image relative to the reference frame, and then spatially transforms each image based on the optical flow results to ensure that all images are spatially aligned. The aligned images are then used to extract shallow features through convolution blocks.

[0061] A high-frequency enhancement module, which separates the input feature map into high-frequency information and low-frequency information through average pooling and maximum pooling layers, quickly capturing high-frequency and low-frequency features of the image;

[0062] A multi-branch fusion module, which uses a multi-branch convolutional fusion structure to process high-frequency and low-frequency information through multi-path convolution combination;

[0063] The image output module aligns the results of multiple high-frequency enhancement sub-networks with the reference frame through simple convolution blocks and fuses them to generate the final HDR image.

[0064] Furthermore, the image output module also includes a tone mapping submodule, which performs tone mapping on the fused image, converting it from a linear domain to a tone mapping domain, and ultimately generates an HDR image with high dynamic range, no noise, and no blur.

[0065] The beneficial effects of the present invention are: the method and system of the present invention repair and enhance multi-exposure images based on a composite optimization model, greatly improving the repair accuracy, and the detail recovery in high-frequency areas such as flame edges and facial contours is significantly better than traditional methods; and the present invention improves the model reasoning efficiency through targeted design, and the reasoning time is lower than that of similar models, thereby ensuring the real-time performance of model reasoning.

[0066] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description and the preceding claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0068] Figure 1 Schematic diagram of the overall process of multi-exposure image restoration and enhancement of the present invention.

[0069] Figure 2 (a), (b), (c), (d) and (e) are images collected at different exposure levels of the same scene;

[0070] Figure 3 for Figure 2 The labels corresponding to the multi-exposure image instances (i.e., the expected image restoration and enhancement results);

[0071] Figure 4 The image is the result obtained after image restoration using the method of the present invention. DETAILED DESCRIPTION

[0072] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, and are not intended to limit the scope of protection of the present invention.

[0073] The main problems that need to be solved by traditional image restoration methods based on image processing and machine learning include: (1) how to uniformly process multi-exposure images to simultaneously achieve denoising, deblurring, and HDR reconstruction; (2) how to effectively preserve and enhance high-frequency details (such as edges and textures); and (3) how to reduce computational costs and improve real-time performance.

[0074] In order to solve the above problems, the present invention proposes a method based on a composite optimization network, which realizes high-quality image restoration and enhancement through joint processing of multi-exposure images, frequency separation enhancement and efficient feature fusion, while significantly reducing computing resource consumption.

[0075] like Figure 1 As shown, Figure 1 : This is a schematic diagram of the overall process of multi-exposure image restoration and enhancement. The multi-exposure image integrated restoration and enhancement method based on a composite optimization network of the present invention includes the following steps:

[0076] Step S1: Image input: multiple images with different exposure times are input. The images are taken in the same scene and arranged from low to high exposure time to ensure that the bright and dark details of the scene are covered. After normalization and gamma correction are performed on each image, they are stitched with the corresponding original image.

[0077] Step S2: Optical flow alignment: Use a pre-trained network to calculate the optical flow between multiple exposure images and estimate the motion vector of each image relative to the reference frame. Based on the optical flow results, each image is spatially transformed to ensure that all images are spatially aligned and eliminate misalignment caused by camera shake or object motion. The aligned images are then used to extract shallow features through convolution blocks, providing high-quality input for subsequent high-frequency enhancement.

[0078] Step S3: High-frequency enhancement: Through the average pooling and maximum pooling layers, the input feature map is separated into high-frequency information and low-frequency information, quickly capturing the high-frequency and low-frequency features of the image;

[0079] Step S4: Multi-branch fusion: A multi-branch convolution fusion structure is used to process high-frequency and low-frequency information separately through multi-path convolution combination;

[0080] Step S5: Output: The results of multiple high-frequency enhancement sub-networks are aligned with the reference frame through a simple convolution block and then fused to generate the final HDR image.

[0081] The above steps will be further described below through specific examples.

[0082] In step S1, this embodiment uses i images with different exposure times as input. These images are taken in the same scene and arranged from low to high exposure time to ensure coverage of both bright and dark details of the scene. After normalization and gamma correction are performed on each image, it is then spliced with the corresponding original image.

[0083] The specific steps of normalization are as follows: suppose there are i images, namely {R1, R2, R3, R4, ..., R i}, according to the exposure value of the image arranged from low to high, R1 is used as a reference, and each image is normalized. The formula is as follows:

[0084]

[0085] where Δt i Represents the exposure time of the i-th image, and then performs gamma correction. The formula is as follows:

[0086]

[0087] Where γ represents the gamma correction parameter. Next, the image data before and after gamma correction are spliced to obtain {I1, I2, I3, I4, ..., I i}, the formula is as follows:

[0088]

[0089] Step S2: Optical flow alignment: Use a pre-trained network to calculate the optical flow between multiple exposure images, estimate the motion vector of each image relative to the reference frame (usually the image with the shortest exposure time), and perform a spatial transformation (such as affine transformation or perspective transformation) on each image based on the optical flow results to ensure that all images are spatially aligned and eliminate misalignment problems caused by camera shake or object motion; the aligned images are used to extract shallow features through convolution blocks to provide high-quality input for the subsequent high-frequency enhancement module.

[0090] Affine transformation and perspective transformation are two common geometric transformations widely used in fields such as image processing, computer vision, and graphics. Affine transformation is a linear transformation that preserves the linearity and proportionality of parallel lines. It includes operations such as translation, rotation, scaling, and shearing. Perspective transformation is a nonlinear transformation that simulates the camera imaging process. It maps points in three-dimensional space onto a two-dimensional plane through central projection and can be used to correct or generate images with perspective distortion.

[0091] In this step, the pre-trained network is used to align the multi-exposure images to solve the image misalignment problem caused by camera shake or object motion; the aligned images are used to extract shallow features through convolution blocks to ensure the spatial consistency of the input images and significantly improve the image restoration and enhancement effects. In this embodiment, i is 5, that is, there are 5 images. Specifically, the {I1, I2, I3, I4, I5} obtained in step S1 are input into the optical flow alignment model, making full use of the information of each image, and finally outputting the aligned and fused feature F align , the formula is as follows:

[0092] F align =f(I1,I2,I3,I4,I5;θ) (4)

[0093] The function f(·) represents the optical flow alignment network model, and θ represents the parameters of this network model. After alignment, the shallow features are extracted and calculated as follows:

[0094] F=Conv(ReLU(Conv 5×5 (F align )));

[0095] ReLU is a nonlinear activation function, which is used to obtain shallow features and input them into the subsequent network structure for further feature extraction and learning.

[0096] Step S3: High-frequency enhancement: The input feature map is separated into high-frequency information and low-frequency information through average pooling and maximum pooling layers. The high-frequency information referred to in this embodiment includes information such as edges and textures, and the low-frequency information includes information such as lighting and contours. This separation method can quickly capture the high-frequency and low-frequency features of the image, providing a basis for subsequent targeted enhancement. This step separates the input feature map through average pooling and maximum pooling layers, and uses a multi-branch structure to achieve targeted processing of high- and low-frequency information to ensure the overall quality of the image.

[0097] In this embodiment, the shallow features F∈R extracted in step 2 are B×H×W×C , the feature separation of high and low frequency information is achieved through the following formula:

[0098] F L =Pooling(F) (5)

[0099] F H =F-Upsample(F L ) (6)

[0100] Among them, Pooling is a downsampling operation, and Upsample is a bilinear interpolation upsampling operation. The low-frequency features are obtained through pooling operation, F H ∈R B×H×W×C It is the high-frequency feature obtained by subtracting the low-frequency feature information from the original feature.

[0101] Step S4: Multi-branch fusion: Using a multi-branch convolution fusion structure, high-frequency and low-frequency information are processed through multi-path convolution combination, and the high-frequency feature F is obtained after the clear extraction. H Afterwards, the self-attention mechanism is used to globally enhance it to make full use of high-frequency information and obtain F H1 ;

[0102] Q H =Conv 1×1 (F H ),KH =Conv 1×1 (F H ),V H =Conv 1×1 (F H );

[0103]

[0104] in is the scaling factor; d is the query vector dimension, which is used to stabilize the attention calculation and prevent the gradient of the Softmax calculation from disappearing. However, fusing features between different frequency domains simply through basic convolution blocks may lead to information loss and insufficient fusion. Therefore, a multi-branch module is adopted to fully fuse high and low frequency features.

[0105] H=MBB(F H1 ) (7)

[0106] L=MBB(MBB(MBB(F L ))) (8)

[0107] Where H and L represent the high-frequency and low-frequency features after dual-path convolution, respectively. MBB stands for a dual-branch convolution architecture. In this embodiment, branch 1 is used to enhance high-frequency details using three 3×3 convolution kernels, while branch 2 focuses on preserving low-frequency contours using a single 1×1 convolution kernel.

[0108] On the basis of the dual branches, the present invention designs an additional dynamic deformable convolution cross branch, namely branch 3, which includes two parts: generating the dynamic convolution kernel offset and generating the deformable convolution output, as shown in the following formula:

[0109] ΔW d =σ(Conv 1×1 (H; L))

[0110] Where ΔW d Denotes the generated dynamic convolution kernel offset, and σ represents the sigmoid function, which maps the input to a range between 0 and 1. Specifically, through unit convolution, the high-frequency features H and low-frequency features L are concatenated and fused, and then constrained using sigmoid to achieve the ability to adaptively adjust the sampling position and address motion blur and edge distortion. Based on the dynamic convolution kernel offset, the deformable convolution output is generated:

[0111]

[0112] Where K is the number of convolution kernels, and p is the feature map coordinate, which corresponds to the sampling point of the convolution kernel. It retains the conventional convolution structure while introducing deformation capability.

[0113] After extracting high-frequency features, the present invention also designs a frequency-aware cross-attention mechanism to further enhance high-frequency features and compensate for low-frequency features. The specific calculation formula is as follows:

[0114] Q H =Conv 1×1 (H),K L =Conv 1×1 (L),V L =Conv 1×1 (L);

[0115]

[0116] in is the scaling factor, and d is the query vector dimension. This is typically used to stabilize attention computations and prevent gradient vanishing in Softmax calculations. By designing a frequency-aware cross-attention module, we establish cross-frequency correlations between high- and low-frequency information, enabling high-frequency features to gain complementary information from the low-frequency domain, thereby enhancing the representational power of high-frequency features.

[0117] Finally, the obtained high-frequency features and low-frequency features are fused, based on which a multi-scale residual gated fusion mechanism is designed. This fusion mechanism constructs a three-branch structure, corresponding to high-frequency features, low-frequency features, and cross-frequency interaction modules. The three branches are weighted and fused to output the final high- and low-frequency fusion features. Branch I mainly performs dilated convolution on high-frequency features to enhance the receptive field of high-frequency features. The calculation formula is as follows:

[0118]

[0119] Where r represents the dilation rate of the dilated convolution, and takes 1, 2, and 3 to correspond to standard convolution, medium receptive field, and large receptive field, respectively, from local features to global features, improving the network learning ability.

[0120] Branch II mainly optimizes low-frequency features and uses unit convolution and spatial attention mechanisms to improve the performance of low-frequency features. The calculation formula is as follows:

[0121] F spatial =SA(Conv 1×1 (L));

[0122] Where SA represents the calculation of the spatial attention mechanism, F spatial It is the optimized low-frequency spatial domain feature.

[0123] Branch III realizes the cross-frequency interactive fusion of high-frequency features and low-frequency features, mainly using a weighted gating mechanism. The calculation formula is as follows:

[0124] G=σ(Conv1×1 (H+L));

[0125] F gate =G·H+(1-G)·L;

[0126] Where G represents the gate weight, which is compressed by the sigmoid function and weighted by the fusion of high and low frequency features in the form of weights, and finally the fused feature F is obtained. gate .

[0127] After completing the calculation of the above three branches, the multi-scale residual gated fusion module will perform weighted fusion on the above three branches and finally output them. The calculation is as follows:

[0128] F final =α·F gate +β·F dilated +γ·F spatial ;

[0129] Among them F final It is the final output of the entire step 4, α+β+γ=1 and is a learnable parameter.

[0130] Step S5: Output: The results of multiple high-frequency enhancement sub-networks are aligned with the reference frame through a simple convolution block and then fused to generate the final HDR image. The calculation formula is as follows:

[0131]

[0132] Where Tanh represents the nonlinear activation function, F align Denotes the reference frame. The multi-branch fusion features are concatenated with the reference frame to obtain a synthetic HDR image.

[0133] During model training, the tone mapping formula is applied to the loss function calculation process. This primarily utilizes logarithmic transformation to alleviate the gradient exploding or vanishing problem caused by extreme brightness values (such as the sun and lights), making backpropagation more stable. The tone mapping formula is as follows:

[0134]

[0135] In this embodiment, μ = 5000, which is used to adjust the image contrast to avoid overexposure while preserving details. The final loss function is thus defined as:

[0136]

[0137] Among them, ‖·‖2 represents the norm operation on ·, and T(H) represent the results of tone mapping between the model output image and the true label image, respectively.

[0138] In order to further verify the practical effect of the method provided by the present invention, a specific example of image acquisition and processing will be provided below to deepen the understanding of the implementation process of the method of the present invention. Figures 2 to 4 As shown, Figure 2 is an example of a multi-exposure image of the present invention, where (a), (b), (c), (d), and (e) are images captured at different exposure levels of the same scene; Figure 3 for Figure 2 The labels corresponding to the multi-exposure image instances in , that is, the expected result images of image restoration and enhancement. Figure 4 This image shows the inpainted result of a multi-exposure image inpainting and enhancement network using the method described in this invention. As can be seen, the inpainted image achieved by this method is essentially consistent with the expected inpainted and enhanced image result, demonstrating that this method significantly improves inpainting accuracy and significantly outperforms traditional methods in recovering detail in high-frequency regions.

[0139] In addition, based on the design concept of the above method, the present invention also provides a multi-exposure image integrated restoration and enhancement system based on a composite optimization network, the system comprising:

[0140] (1) Image input and processing module: used to receive external images, which are taken in the same scene and arranged in order of exposure time from low to high to ensure that the bright and dark details of the scene are covered. After normalization and gamma correction are performed on each image, it is spliced with the corresponding original image;

[0141] (2) Optical flow alignment module: Calculate the optical flow between multiple exposure images using a pre-trained network, estimate the motion vector of each image relative to the reference frame, and perform spatial transformation on each image based on the optical flow results to ensure that all images are spatially aligned. After alignment, shallow features are extracted from the images through convolution blocks.

[0142] (3) High-frequency enhancement module: Through the average pooling and maximum pooling layers, the input feature map is separated into high-frequency information and low-frequency information, quickly capturing the high-frequency and low-frequency features of the image;

[0143] (4) Multi-branch fusion module: adopts a multi-branch convolution fusion structure to process high-frequency and low-frequency information through multi-path convolution combination;

[0144] (5) Image output module: The results of multiple high-frequency enhancement sub-networks are aligned with the reference frame through a simple convolution block and then fused to generate the final HDR image.

[0145] The image output module also includes a tone mapping submodule, which performs tone mapping on the fused image, converting it from the linear domain to the tone mapping domain, and finally generates an HDR image with high dynamic range, no noise, and no blur.

[0146] It should be noted that, in this specification, the devices disclosed in the embodiments are described in a relatively simple manner because they correspond to the methods disclosed in the embodiments. For relevant details, please refer to the description of the methods.

[0147] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0148] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. An integrated multi-exposure image restoration and enhancement method based on a composite optimization network, characterized by: The method comprises the following steps: Step S1: Image input: multiple images with different exposure times are input. The images are taken in the same scene and arranged from low to high exposure time to ensure that the bright and dark details of the scene are covered. After normalization and gamma correction are performed on each image, they are stitched with the corresponding original image. Step S2: Optical flow alignment: Use a pre-trained network to calculate the optical flow between multiple exposure images and estimate the motion vector of each image relative to the reference frame. Based on the optical flow results, each image is spatially transformed to ensure that all images are spatially aligned and eliminate misalignment caused by camera shake or object motion. The aligned images are then used to extract shallow features through convolution blocks, providing high-quality input for subsequent high-frequency enhancement. Step S3: High-frequency enhancement: Through the average pooling and maximum pooling layers, the input feature map is separated into high-frequency information and low-frequency information, quickly capturing the high-frequency and low-frequency features of the image; Step S4: Multi-branch fusion: A multi-branch convolution fusion structure is used to process high-frequency and low-frequency information separately through multi-path convolution combination; Step S5: Output: The results of multiple high-frequency enhancement sub-networks are aligned with the reference frame through a simple convolution block and then fused to generate the final HDR image.

2. The multi-exposure image integrated restoration and enhancement method based on a composite optimization network according to claim 1, characterized in that: In step S1, suppose there are i images, namely {R1, R2, R3, R4, ..., R i }, according to the exposure value of the image arranged from low to high, R1 is used as a reference, and each image is normalized. The formula is as follows: where Δt i Represents the exposure time of the i-th image, and then performs gamma correction. The formula is as follows: Where γ represents the gamma correction parameter. Next, the image data before and after gamma correction are spliced to obtain {I1, I2, I3, I4, ..., I i }, the formula is as follows:

3. The multi-exposure image integrated restoration and enhancement method based on a composite optimization network according to claim 2, characterized in that: In the step S2, {I1, I2, I3, I4, ..., I i } is input into the optical flow alignment model, making full use of the information of each image, and finally outputting the aligned fusion feature F align , the formula is as follows: F align =f(I1,I2,I3,I4,……,I i ;θ) (4) The function f(·) represents the optical flow alignment network model, and θ represents the parameters of this network model. After alignment, the shallow features are extracted and calculated as follows: F=Conv 1×1 (ReLU(Conv 3×3 (F align )),ReLU(Conv 5×5 (F align ))); Among them, ReLU is a nonlinear activation function, Conv 1×1 、Conv 3×3 and Conv 5×5 They represent convolution operations with kernel sizes of 1, 3, and 5, respectively. Using the convolution operation results of different kernel sizes, we can obtain information of different scales in shallow features, and then realize feature fusion through unit convolution, which can be input into the subsequent network structure for further feature extraction and learning.

4. The multi-exposure image integrated restoration and enhancement method based on a composite optimization network according to claim 3, characterized in that: In step S3, the shallow features F∈R extracted in step 2 are B×H×W×C , the feature separation of high and low frequency information is achieved through the following formula: F L =Pooling(F) (5) F H =F-Upsample(F L ) (6) Among them, Pooling is a downsampling operation, and Upsample is a bilinear interpolation upsampling operation. The low-frequency features are obtained through pooling operation, F H ∈R B×H×W×C It is the high-frequency feature obtained by subtracting the low-frequency feature information from the original feature.

5. The multi-exposure image integrated restoration and enhancement method based on a composite optimization network according to claim 4, characterized in that: In step S4, after obtaining the clearly extracted high-frequency features F H Afterwards, the self-attention mechanism is used to globally enhance it to make full use of high-frequency information and obtain F H1 : Q H =Conv 1×1 (F H ),K H =Conv 1×1 (F H ),V H =Conv 1×1 (F H ) in is the scaling factor; d is the query vector dimension, which is used to stabilize the attention calculation and prevent the Softmax calculation gradient from disappearing. H1 With F L Based on this, a multi-branch structure is adopted to fully integrate high and low frequency features: H=MBB(F H1 ) (7) L=MBB(MBB(MBB(F L ))) (8) In the above formula, H and L represent the high-frequency features and low-frequency features after dual-path convolution processing, respectively. MBB represents a dual-branch convolution structure, where branch 1 is used to enhance high-frequency details and branch 2 is used to preserve contours. The specific formula is as follows: Y1=Conv 3×3 (X) Y2=Conv 3×3 (Conv 3×3 (Conv 3×3 (X))) Where Y1 and Y2 represent the output of branch 1 and branch 2 respectively, Conv 3×3 Represents a convolution operation with a kernel size of 3.

6. The multi-exposure image integrated restoration and enhancement method based on a composite optimization network according to claim 5, characterized in that: On the basis of the dual branches, an additional dynamic deformable convolution cross branch is designed, namely branch 3. Branch 3 includes two parts: generating dynamic convolution kernel offset and generating deformable convolution output, as shown in the following formula: ΔW d =σ(Conv 1×1 (H;L)); Where ΔW d Represents the generated dynamic convolution kernel offset, σ represents the sigmoid function, which maps the input to between 0 and 1. That is, through unit convolution, the high-frequency feature H and the low-frequency feature L are spliced and fused, and then sigmoid is used for constraint to achieve the ability to adaptively adjust the sampling position and solve motion blur and edge distortion. Based on the dynamic convolution kernel offset, the deformable convolution output is generated as shown in the following formula: Where K is the number of convolution kernels, and p is the feature map coordinate, which corresponds to the sampling point of the convolution kernel. It retains the conventional convolution structure while introducing deformation capability, thereby making the convolution operation have a larger receptive field, expanding the range of local features, and having stronger global modeling capabilities.

7. The multi-exposure image integrated restoration and enhancement method based on a composite optimization network according to claim 6, characterized in that: By setting up a frequency-aware cross-attention mechanism, it is used to further enhance high-frequency features and compensate for low-frequency features. The specific calculation formula is as follows: Q H =Conv 1×1 (H),K L =Conv 1×1 (L),V L =Conv 1×1 (L); in is the scaling factor; d is the query vector dimension, which is used to stabilize the attention calculation and prevent the gradient of the Softmax calculation from disappearing. The frequency-aware cross-attention mechanism uses high-frequency features to generate the query vector and low-frequency features to generate the key vector and value vector. Then, attention calculation is performed and feature fusion is performed at the end to achieve mutual compensation and learning of high- and low-frequency features.

8. The multi-exposure image integrated restoration and enhancement method based on a composite optimization network according to claim 7, characterized in that: The high-frequency and low-frequency features are fused using a multi-scale residual gated fusion mechanism. The fusion mechanism includes branches I, II, and III, which correspond to high-frequency features, low-frequency features, and cross-frequency interactions, respectively. The three branches are weighted and fused to output the final high- and low-frequency fusion features. Branch I is used to perform dilated convolution on high-frequency features to enhance the receptive field of high-frequency features. The calculation formula is as follows: Where r represents the dilation ratio of the dilated convolution, and values of 1, 2, and 3 correspond to standard convolution, medium receptive field, and large receptive field, respectively, improving the network learning ability from local features to global features. The branch II is used to optimize the low-frequency features and adopts the unit convolution and spatial attention mechanism to improve the performance of low-frequency features. The calculation formula is as follows: F spatial =SA(Conv 1×1 (IT)); Where SA represents the calculation of the spatial attention mechanism, F spatial It is the optimized low-frequency spatial domain feature; The branch III is used for cross-frequency interactive fusion of low-frequency features, which is implemented using a weighted gating mechanism. The calculation formula is as follows: G=σ(Conv 1×1 (H+L)); F gate =G·H+(1-G)·L; Where G represents the gate weight, which is compressed by the sigmoid function and weighted by the fusion of high and low frequency features in the form of weights, and finally the fused feature F is obtained. gate ; After completing the calculation of the above three branches, the above three branches are weighted fused and finally output. The calculation is as follows: F final =α·F gate +β·F dilated +γ·F spatial ; Among them F final It is the final output of the entire step 4, α+β+γ=1 and is a learnable parameter.

9. An integrated multi-exposure image restoration and enhancement system based on a composite optimization network, characterized by: The system includes an image input and processing module, which is used to receive external images. The images are taken in the same scene with exposure times arranged from low to high to ensure coverage of the bright and dark details of the scene. After normalization and gamma correction of each image, they are spliced with the corresponding original image; The optical flow alignment module uses a pre-trained network to calculate the optical flow between multiple exposure images, estimates the motion vector of each image relative to the reference frame, and then spatially transforms each image based on the optical flow results to ensure that all images are spatially aligned. The aligned images are then used to extract shallow features through convolution blocks. A high-frequency enhancement module, which separates the input feature map into high-frequency information and low-frequency information through average pooling and maximum pooling layers, quickly capturing high-frequency and low-frequency features of the image; A multi-branch fusion module, which uses a multi-branch convolutional fusion structure to process high-frequency and low-frequency information through multi-path convolution combination; The image output module aligns the results of multiple high-frequency enhancement sub-networks with the reference frame through simple convolution blocks and fuses them to generate the final HDR image.

10. The multi-exposure image integrated restoration and enhancement system based on a composite optimization network according to claim 9, characterized in that: The image output module also includes a tone mapping submodule, which performs tone mapping on the fused image, converting it from a linear domain to a tone mapping domain, and ultimately generates an HDR image with high dynamic range, no noise, and no blur.

Images