Dynamic multi-exposure light field image fusion method based on structural consistency detection

Abstract

The present invention discloses a dynamic multi-exposure light field image fusion method based on structural consistency detection. The method performs high-order singular value decomposition on the angular dimension of the registration light field image corresponding to the light field image in the dynamic multi-exposure light field image sequence to obtain a multi-exposure main base sequence and a multi-exposure non-main base sequence; the multi-exposure main base sequence is input into a main base fusion module based on structural consistency detection to obtain a fused main base; the multi-exposure non-main base sequence is input into the remaining baseband fusion module based on mask partitioning to obtain a fused non-main base; a fused baseband tensor is obtained according to the fused main base and all the fused non-main bases, and is reconstructed by combining the fused baseband tensor and the orthogonal factor matrix of two angular dimensions during high-order singular value decomposition to restore the light field structure, and then the angle information is restored to obtain a fused light field image containing complete angle information; the advantage is that the method can obtain a high dynamic range fused light field image with good spatial quality and angular quality.

Description

Technical Field

[0001] The present invention relates to a multi-exposure light field image fusion technology, and in particular to a dynamic multi-exposure light field image fusion method based on structure consistency detection. Background Art

[0002] A light field describes the amount of light flowing through every point in free space in any direction and is a complete representation of a collection of light rays in a three-dimensional world. Light field imaging can simultaneously record the position and direction of light rays in space, and has effective applications in digital refocusing, depth estimation, and multi-view imaging. The advent of light field cameras has promoted the development of light field imaging, while also bringing about further human demands for capturing and perceiving the physical world. Limited by factors such as light characteristics and sensor physical parameters, the rich information present in daily life can be presented as a colorful picture in the visual sense. However, due to the limitations of hardware equipment, the dynamic range of a light field camera's single image capture is much smaller than that of a natural scene, resulting in overexposure or underexposure in some areas.

[0003] To address the technical challenges of light field cameras, most approaches use multi-exposure fusion to fuse a set of images containing scene information at different exposure levels into a single fused image. However, real-world scenes often contain moving objects, which can cause artifacts and blur in the fused image. Currently, research on multi-exposure light field image fusion in dynamic scenes is relatively limited. Therefore, it is crucial to develop a dynamic multi-exposure light field image fusion method. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a dynamic multi-exposure light field image fusion method based on structural consistency detection, which can obtain a high dynamic range fused light field image with good spatial quality and angular quality.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a dynamic multi-exposure light field image fusion method based on structural consistency detection, characterized by comprising the following steps:

[0006] Step 1: Obtain M light field images with different exposure levels taken at the same spatial position, and construct a dynamic multi-exposure light field image sequence from the M light field images with different exposure levels, denoted as {L i ∈R U×V×H×W×C |1≤i≤M}; where M>1, L i represents the light field image of the i-th exposure level, also the i-th light field image, R represents a real number set, L i The spatial resolution is W×H, the angular resolution is V×U, the number of color channels is C, 1≤i≤M;

[0007] Step 2:i ∈R U×V×H×W×C The sub-aperture images of each light field image in |1≤i≤M} are aligned to the center view to obtain the corresponding registered light field image. i The corresponding registered light field image is denoted as L A(i) Among them, L A(i) ∈R U ×V×H×W×C , L A(i) The spatial resolution is W×H, the angular resolution is V×U, and the number of color channels is C;

[0008] Step 3: Perform high-order singular value decomposition on the angular dimension of each registered light field image to obtain the corresponding baseband tensor. The first component of the baseband tensor is used as the main basis, and the remaining basebands except the first component of the baseband tensor are used as non-main basis. A(i) The principal basis in the corresponding baseband tensor is denoted as P B(i) , the set of principal bases in the baseband tensors corresponding to all registered light field images {P B(i) |1≤i≤M} as the multi-exposure main base sequence, and L A(i) The kth non-primary basis in the corresponding baseband tensor is denoted as P O(i),k , the set of the kth non-primary basis at the same position in the basis band tensor corresponding to all registered light field images {P O(i),k |1≤i≤M} is used as the k-th multi-exposure non-primary basis sequence, and there are M non-primary basis in the k-th multi-exposure non-primary basis sequence; where 1≤k≤K, K represents the number of basebands except the first component in the baseband tensor corresponding to each registered light field image, K=V×U-1;

[0009] Step 4: Input the multi-exposure main base sequence into the main base fusion module based on structure consistency detection to obtain the fused main base, denoted as P BB ; Among them, after obtaining the fusion main base P BB In the process of obtaining the structural consistency detection binary image of each main base in the multi-exposure main base sequence;

[0010] Step 5: Input the k-th multi-exposure non-primary base sequence into the remaining baseband fusion modules based on mask partitioning to obtain the k-th fused non-primary base, denoted as P OB,k ; Among them, when dividing the mask, the structural consistency detection binary image corresponding to the main basis is used as the mask;

[0011] Step 6: According to P BB and the set of all fused non-primary bases {P OB,k |1≤k≤K}, and the fused baseband tensor is obtained, which is recorded as Then combine The orthogonal factor matrices of the two angular dimensions when the angular dimension of the registered light field image is decomposed by high-order singular value are reconstructed to restore the light field structure. The obtained fused light field image is recorded as

[0012] Step 7: Right The angle information of the image is restored to obtain a fused light field image containing complete angle information, which is recorded as L H .

[0013] The specific process of step 2 is: using the optical flow estimation method to calculate {L i ∈R U×V×H×W×C The disparity between the central sub-aperture image and each non-central sub-aperture image of each light field image in |1≤i≤M} is calculated; then, the non-central sub-aperture images of each light field image are mapped onto the central sub-aperture image according to the corresponding disparity relationship to achieve central view alignment, thereby obtaining the corresponding registered light field image; wherein the non-central sub-aperture images are the remaining sub-aperture images except the central sub-aperture image.

[0014] The optical flow estimation method is specifically a SIFT flow optical flow estimation method.

[0015] In step 3, P B(i) and P O(i),k The acquisition process is:

[0016] Step 3.1: L A(i) Perform high-order singular value decomposition, the decomposition formula is: L A(i) =S i ×1O (1) ×2O (2) ×3O (3) ×4O (4) ×5O (5) ; Among them, S i Indicates L A(i) The core tensor obtained after high-order singular value decomposition, S i ∈R U'×V'×H'×W'×C' , U' and V', H', W', C' are S i The five dimensions of O (1) and O (2) is the orthogonal factor matrix of two angle dimensions, O (1) ∈R U×U' , O (2) ∈R V×V' , O (3) and O (4) is the orthogonal factor matrix of two spatial dimensions, O (3) ∈R H×H' , O (4) ∈R W×W', O (5) Represents the orthogonal factor matrix of the color channel dimension, O (5) ∈R C×C' ;

[0017] Step 3.2: Place L A(i) =S i ×1O (1) ×2O (2) ×3O (3) ×4O (4) ×5O (5) Convert to Here, the superscript "T" indicates the transpose of a vector or matrix;

[0018] Step 3.3: Based on the calculation of L A(i) The baseband tensor obtained by high-order singular value decomposition of the angular dimension is denoted as P i , P i The first component in the main basis P B(i) , P i The kth baseband except the first component is taken as the kth non-primary basis P O(i),k .

[0019] In step 4, the main base fusion module based on structure consistency detection is used to process the multi-exposure main base sequence to obtain the fused main base P BB The process is:

[0020] Step 4.1: Segment each primary basis in the multi-exposure primary basis sequence into multiple overlapping color blocks in RGB space. Segment the overlapping color blocks using a sliding window with a step size of 2 pixels across the primary basis. The size of the color blocks is 24×24.

[0021] Step 4.2: Select a reference master basis from the multi-exposure master basis sequence and record the reference master basis as P r ; Then use the intensity mapping function and the reference main basis P r To generate a multi-exposure virtual main base sequence, the i-th virtual main base in the multi-exposure virtual main base sequence is recorded as P X(i) ; Then traverse each main basis in the multi-exposure main basis sequence and define the currently traversed main basis as the current main basis; wherein the reference main basis is {L i ∈R U×V×H×W×C The principal basis in the baseband tensor corresponding to the registered light field image corresponding to the light field image with the best exposure in |1≤i≤M};

[0022] Step 4.3: Set the current main base to P B(i) , P B(i) The jth color block in is denoted as xi,j , and x i,j Expressed as the signal strength component c i,j , signal structure component s i,j and the average intensity component l i,j Three parts, Among them, when segmenting overlapping color blocks, a sliding window is used with a step size of 2 pixels in P B(i) Segmentation is achieved by sliding in the middle. The size of the color block is 24×24, 1≤j≤J, J represents P B(i) The total number of colored blocks in c i,j and l i,j Both are scalars, s i,j is a vector, and “||||” is the L2 norm calculation symbol. Represents x i,j The average value of all pixel values in , "·" is the dot multiplication operator,

[0023] Step 4.4: P B(i) Partition and set P B(i) Divided into overexposed area and non-overexposed area; then according to P B(i) The over-exposure area and the non-over-exposure area are obtained. B(i) The corresponding over-exposed binary image is recorded as

[0024] Step 4.5: Use the perceptual hashing algorithm to obtain P B(i) The corresponding first structural consistency region detection binary image is recorded as The acquisition process is: calculate the reference main basis P r Each colored block in P B(i) The Hamming distance of the hash value of the corresponding color block in the reference base P r The j-th color block x in r,j With P B(i) The j-th color block x in i,j The Hamming distance of the hash value is recorded as δ i,j , δ i,j =F PHA (x r,j ,x i,j ); Then according to the reference main base P r All the colored blocks in P B(i) The Hamming distance of the hash value of the corresponding color block in is obtained The number of pixels in each row is the same as P B(i) The number of colored blocks in each row is equal. The number of pixels in each column is the same as P B(i) The number of color blocks in each column is equal. If the reference base P rA colored block in P B(i) The Hamming distance of the hash value of the corresponding color block in is less than or equal to the threshold th1, then The pixel value of the corresponding pixel point in the reference position is 0; if the reference base P r A colored block in P B(i) The Hamming distance of the hash value of the corresponding color block in is greater than the threshold th1, then The pixel value of the corresponding pixel point in the P is 1; where 1≤j≤J, J represents P B(i) The total number of colored blocks in δ i,j ∈[0,64],F PHA () indicates the perceptual hash algorithm operation, th1 = 5;

[0025] Step 4.6: Use the normalized cross-correlation algorithm to obtain P B(i) The corresponding second structure consistency region detection binary image is recorded as The acquisition process is: calculate the reference main basis P r The signal structure component of each color block in P B(i) The inner product of the signal structure component of the corresponding color block in the reference base P r The j-th color block x in r,j The signal structure component s r,j With P B(i) The j-th color block x in i,j The signal structure component s i,j The inner product is denoted as θ i,j , Then according to the reference main base P r The signal structure components of all color blocks in P B(i) The inner product of the signal structure component of the corresponding color block in The number of pixels in each row is the same as P B(i) The number of colored blocks in each row is equal. The number of pixels in each column is the same as P B(i) The number of color blocks in each column is equal. If the reference base P r The signal structure component of a color block in P B(i) The inner product of the signal structure component of the corresponding color block is greater than or equal to the threshold constant th2, then The pixel value of the corresponding pixel point in the reference position is 0; if the reference base P r The signal structure component of a color block in P B(i) The inner product of the signal structure component of the corresponding color block is less than the threshold constant th2, then The pixel value of the corresponding pixel point in θ is 1; i,j ∈[-1,1],lr,j Indicates the reference primary base P r The j-th color block x in r,j The average intensity component, ε is an infinitesimal number used to prevent the denominator from being 0, th2 = 0.8;

[0026] Step 4.7: Calculate the reference principal basis P r The average intensity component of each color block in P X(i) The average intensity difference between the average intensity components of the corresponding color blocks in the reference base P r The j-th color block x in r,j The average intensity component l r,j With P X(i) The j-th color block x in X(i),j The average intensity component l X(i),j The average intensity difference between i,j ,ζ i,j =||l r,j -l X(i),j ||; then according to the reference main base P r The average intensity component of all color blocks in P X(i) The average intensity difference between the average intensity components of the corresponding color blocks in The number of pixels in each row is the same as P B(i) The number of colored blocks in each row is equal. The number of pixels in each column is the same as P B(i) The number of color blocks in each column is equal. If the reference base P r The average intensity component of a color block in P X(i) The average intensity difference between the average intensity components of the corresponding color blocks in is greater than or equal to the threshold constant th3, then The pixel value of the corresponding pixel point in the reference position is 0; if the reference base P r The average intensity component of a color block in P X(i) The average intensity difference between the average intensity components of the corresponding color blocks in is less than the threshold constant th3, then The pixel value of the pixel at the corresponding position in is 1; where th3 = 0.1;

[0027] Step 4.8: According to Get P B(i) The structural consistency detection binary map is denoted as Z i ,

[0028] Step 4.9: Determine P B(i) Whether each colored block in P belongs to the structural consistency area, B(i) The j-th color block x ini,j , if Z i Zhong and x i,j The pixel value of the pixel at the corresponding position is 0, then x i,j Belongs to the structural consistency region; if Z i Zhong and x i,j The pixel value of the pixel at the corresponding position is 1, then x i,j It belongs to the area of structural inconsistency;

[0029] Step 4.10: For P B(i) The colored blocks in the structural inconsistency area are used to X(i) The corresponding colored blocks in are replaced to achieve compensation, that is: assuming x i,j Belong to the structural inconsistency area, then use P X(i) The j-th color block x in X(i),j Replace x i,j ; will be P B(i) The compensation main basis obtained after compensating all the color blocks belonging to the structural inconsistency area is recorded as P' B(i) ;

[0030] Step 4.11: Use the next traversed main basis in the multi-exposure main basis sequence as the current main basis, and then return to step 4.3 to continue executing until the compensated main basis corresponding to each main basis in the multi-exposure main basis sequence is obtained;

[0031] Step 4.12: Based on the color blocks at the same position in the compensation main basis corresponding to all the main basis in the multi-exposure main basis sequence, that is, a total of M color blocks, a corresponding fused image block is reconstructed. i,j The fused image block reconstructed from the M color blocks of |1≤i≤M} is recorded as Expressed as signal strength component Signal structure component and the average intensity component Three parts, Among them, x' i,j Represents P' B(i) The jth color block in and are scalars, is a vector, max() is the maximum value function. express The average signal structure component of ω1() is a weighted function, and ω1() is expressed by an exponential function. β is the exponential parameter, β = 4, μ' i Represents P'B(i) The global mean of ω2() is the weighting function, and ω2() adopts a two-dimensional normal distribution. exp() represents the exponential function with the natural base e as the base, e=2.71…,μ c 、l c , σ x , σ y are the parameters of the two-dimensional normal distribution, σ x =0.2,σ y =0.5, μ c =0.5, l c =0.5, c' i,j represents x' i,j The signal strength component, s' i,j represents x' i,j The signal structure component, l' i,j represents x' i,j The average intensity component of represents x' i,j The average value of all pixel values in ;

[0032] Step 4.13: Press P B(i) The reverse process of segmenting into overlapping colored blocks is to All fused image blocks are recombined into a main basis as the fusion main basis P BB ; Among them, the average value of the pixel values of the pixels at the same position in the overlapping area during reorganization is used as the final pixel value after reorganization.

[0033] The specific process of step 4.4 is as follows:

[0034] Step 4.4.1: Place P B(i) The color space is converted from RGB space to HSV model; then P is converted to HSV model under HSV model. B(i) Divide into multiple overlapping blocks; then compare P B(i) The size of the region division index of each overlapping block in the image is compared with the set division threshold. If the region division index is greater than or equal to the set division threshold, the overlapping block is considered to belong to the overexposed area; if the region division index is less than the set division threshold, the overlapping block is considered to belong to the non-overexposed area. When segmenting overlapping blocks, a sliding window is used with a step size of 2 pixels in P B(i) Segmentation is achieved by sliding in the middle. The size of the overlapping block is 24×24. The region division index is equal to the number of pixels in the overlapping block with a V component value greater than 240 divided by the total number of pixels in the overlapping block. The segmentation threshold is set to be the product of 0.8 and the total number of pixels in the overlapping block.

[0035] Step 4.4.2: Let Indicates P B(i) The corresponding over-exposed binary image, The number of pixels in each row is the same as P B(i) The number of overlapping blocks in each row is equal, The number of pixels in each column is the same as P B(i) The number of overlapping blocks in each column is equal. If P B(i) One of the overlapping blocks belongs to the overexposed area, then The pixel value of the corresponding pixel in the B(i) One of the overlapping blocks belongs to the non-overexposed area, then The pixel value of the pixel at the corresponding position in is 0.

[0036] In step 5, the remaining baseband fusion modules based on mask division are used to process the k-th multi-exposure non-primary base sequence to obtain the k-th fused non-primary base sequence P OB,k The process is:

[0037] Step 5.1: Segment each non-primary basis in the k-th multi-exposure non-primary basis sequence into multiple overlapping color blocks in RGB space. Segment the overlapping color blocks using a sliding window with a step size of 2 pixels across the non-primary basis. The size of the color blocks is 24×24.

[0038] Step 5.2: Classify all color blocks in each non-main base in the k-th multi-exposure non-main base sequence into mask areas and non-mask areas. O(i),k , Z i As a mask to P O(i),k The process of classifying each color block in P is as follows: O(i),k The j-th color block x in i,k,j , if Z i Zhong and x i,k,j The pixel value of the pixel at the same position is 1, then x i,k,j Belongs to the mask area; if Z i Zhong and x i,k,j The pixel value of the pixel at the same position is 0, so it is considered that x i,k,j Belongs to the non-mask area;

[0039] Step 5.3: Get the kth fused non-primary basis P OB,k The non-mask area is denoted as P Os,k , Among them, max() is the maximum value function, and "·" is the dot multiplication operator. Indicates Z i The complementary graph obtained by performing the inverse operation;

[0040] Step 5.4: Get the kth fused non-primary basis POB,k The mask area is denoted as P Od,k , the acquisition process is:

[0041] Step 5.4.1: For each color block in the mask area in the non-primary basis in the k-th multi-exposure non-primary basis sequence, use P X(i) The corresponding color blocks in P are replaced to achieve compensation, that is: O(i),k , assuming x i,k,j Belongs to the mask area, then use P X(i) The j-th color block x in X(i ) ,j Replace x i,k,j ; will be P O(i),k The compensated non-primary basis obtained after compensating all the color blocks belonging to the mask area is recorded as P' O(i),k ;

[0042] Step 5.4.2: Reconstruct a corresponding fused image block based on the color blocks at the same position in the compensated non-main base corresponding to all non-main bases in the k-th multi-exposure non-main base sequence, that is, a total of M color blocks. i,k,j The fused image block reconstructed from the M color blocks of |1≤i≤M} is recorded as Expressed as signal strength component Signal structure component and the average intensity component Three parts, Among them, x' i,k,j Represents P' O(i),k The jth color block in Obtained as in step 4.12;

[0043] Step 5.4.3: Press P O(i),k The reverse process of segmenting into overlapping colored blocks is to All fused image blocks are reorganized into a mask region P Od,k ; Among them, the average value of the pixel values of the pixels at the same position in the overlapping area during reorganization is used as the final pixel value after reorganization;

[0044] Step 5.5: Get the kth fused non-primary basis P OB,k , P OB,k =P Os,k +P Od,k .

[0045] In step 6,

[0046] The specific process of step 7 is as follows: i ∈R U×V×H×W×CThe disparity between the central sub-aperture image of the light field image with the best exposure in |1≤i≤M} and each non-central sub-aperture image is converted into The non-central sub-aperture images aligned with the central views of the image are mapped to their corresponding positions to restore the angle information, and L H .

[0047] Compared with the prior art, the advantages of the present invention are:

[0048] 1) Taking into account that the presence of moving objects in dynamic scenes will cause distortions such as artifacts and blur in the fused light field, the method of the present invention uses a perceptual hash algorithm and a normalized cross-correlation algorithm to perform structural consistency detection, and applies them to the fusion processing of multi-exposure main base sequence and multi-exposure non-main base (other baseband) sequences, respectively. Based on this, a high dynamic range light field with good spatial and angular quality can be obtained.

[0049] 2) Taking into account that light field data contains a lot of redundant information, the method of the present invention uses high-order singular value decomposition to reduce the dimensionality of the high-dimensional 4D light field data, realizing the conversion of 4D light field to 2D image, which reduces the computational complexity to a certain extent. At the same time, the method uses optical flow estimation and parallax rendering to achieve the alignment of the central view of the light field, effectively alleviating the baseband blur after dimensionality reduction caused by the differences between light field images under different viewing angles.

[0050] 3) The method of the present invention takes into account the importance of multi-exposure non-main base (rest baseband) sequences in light field information. Different from directly using any other baseband information after high-order singular value decomposition of any exposure light field as the other baseband information for reconstructing the light field, a mask-based rest baseband fusion method is adopted here to effectively fuse the high-frequency information of the multi-exposure light field image sequence, which can effectively improve the imaging quality of the fused light field. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flowchart of the overall implementation of the method of the present invention;

[0052] Figure 2 Schematic diagram of the processing process of the main baseband fusion module based on structure consistency detection and the remaining baseband fusion modules based on mask partitioning in the method of the present invention;

[0053] Figure 3a This is the underexposed light field image in the fourth test light field image sequence. Here, the central sub-aperture image (including the local magnified image) and its corresponding EPI magnified image are shown;

[0054] Figure 3bThis is the medium-exposure light field image in the fourth test light field image sequence. Here, the central sub-aperture image (including the local magnification image) and its corresponding EPI magnification image are shown;

[0055] Figure 3c This is the overexposed light field image in the fourth test light field image sequence. Here, the central sub-aperture image (including the local magnified image) and its corresponding EPI magnified image are shown;

[0056] Figure 3d This is a fused light field image obtained by processing the fourth test light field image sequence using the method of Mertens et al. Here, the fusion result of the central sub-aperture image (including the local magnification image) and its corresponding EPI magnification image are shown;

[0057] Figure 3e This is the fused light field image obtained by processing the fourth test light field image sequence using the method of Li et al. (GFF). Here, the fusion result of the central sub-aperture image (including the local magnification image) and its corresponding EPI magnification image are shown;

[0058] Figure 3f This is the fused light field image obtained by processing the fourth test light field image sequence using the method of Liu et al. (DSIFT-MEF). Here, the fusion results of the central sub-aperture image (including the local magnification image) and its corresponding EPI magnification image are shown;

[0059] Figure 3g This is a fused light field image obtained by processing the fourth test light field image sequence using the method of Hayat et al. The fusion results of the central sub-aperture image (including the local magnification image) and its corresponding EPI magnification image are shown here;

[0060] Figure 3h This is a fused light field image obtained by processing the fourth test light field image sequence using the method of Li et al. (FMSSPD-MEF). Here, the fusion results of the central sub-aperture image (including the local magnification image) and its corresponding EPI magnification image are shown;

[0061] Figure 3i This is a fused light field image obtained by processing the fourth test light field image sequence using the method of Li et al. (MESPD). Here, the fusion results of the central sub-aperture image (including the local magnification image) and its corresponding EPI magnification image are shown;

[0062] Figure 3j This is a fused light field image obtained by processing the fourth test light field image sequence using the method of the present invention. The fusion result of the central sub-aperture image (including the local magnification image) and its corresponding EPI magnification image are shown here.

[0063] Figure 4a is the central sub-aperture image of the underexposed light field image in the 22nd test light field image sequence;

[0064] Figure 4b is the central sub-aperture image of the medium-exposure light field image in the 22nd test light field image sequence;

[0065] Figure 4c is the central sub-aperture image of the overexposed light field image in the 22nd test light field image sequence;

[0066] Figure 4d is the depth map of the underexposed light field image in the 22nd test light field image sequence;

[0067] Figure 4e is the depth map of the medium-exposure light field image in the 22nd test light field image sequence;

[0068] Figure 4f is the depth map of the overexposed light field image in the 22nd test light field image sequence;

[0069] Figure 4g The depth map of the fused light field image obtained by processing the 22nd test light field image sequence using the method of Mertens et al.

[0070] Figure 4h is the depth map of the fused light field image obtained by processing the 22nd test light field image sequence using the method of Li et al. (GFF);

[0071] Figure 4i The depth map of the fused light field image obtained by processing the 22nd test light field image sequence using the method of Liu et al. (DSIFT-MEF);

[0072] Figure 4j The depth map of the fused light field image obtained by processing the 22nd test light field image sequence using the method of Hayat et al.

[0073] Figure 4k The depth map of the fused light field image obtained by processing the 22nd test light field image sequence using the method of Li et al. (FMSSPD-MEF);

[0074] Figure 4l The depth map of the fused light field image obtained by processing the 22nd test light field image sequence using the method of Li et al. (MESPD);

[0075] Figure 4mIt is a depth map of the fused light field image obtained by processing the 22nd test light field image sequence using the method of the present invention. DETAILED DESCRIPTION

[0076] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0077] With the continuous advancement of computer and optoelectronic technologies, light field imaging has gradually become a research hotspot in computer vision and optical imaging. To a certain extent, light field imaging technology overcomes the problem of traditional imaging techniques ignoring light angle information, bringing innovation to imaging technology and providing the theoretical foundation for the birth of light field cameras. The advent of light field cameras has promoted the development of light field imaging while also fostering a deeper demand for capturing and perceiving the physical world. Since the invention of light field cameras, improving their imaging capabilities has been a constant goal for researchers. Limited by factors such as light characteristics and sensor physical parameters, light field cameras suffer from insufficient dynamic range, making it difficult to develop new light field cameras to improve their imaging quality. Currently, there are two approaches to expanding the field of view and dynamic range of light field camera imaging. The first approach focuses on hardware devices, aiming to improve imaging quality by designing specialized imaging equipment. However, these approaches often suffer from complex designs and portability issues. Therefore, another approach is often adopted: designing appropriate imaging methods based on computational photography to improve the quality of light field camera imaging. To address this issue, from the perspective of the high-dimensional characteristics of light field data and motion area detection, the present invention proposes a dynamic multi-exposure light field image fusion method based on structural consistency detection. The content of the method of the present invention is mainly divided into three parts. First, the multi-exposure light field image sequences are aligned based on the central view to eliminate the differences between different perspectives, and further high-order singular value decomposition is performed to obtain a multi-exposure main basis sequence and a multi-exposure non-main basis sequence; secondly, for the multi-exposure main basis sequence, after partitioning it, the perceptual hash algorithm and the normalized cross-correlation method are combined to detect the structural inconsistency area, and the fused main basis with artifact removal is obtained by combining the image block decomposition method; thirdly, for the multi-exposure non-main basis sequence, the structural consistency area detection binary map obtained in the multi-exposure main basis sequence is used as the mask map, and the multi-exposure non-main basis sequence is divided into mask area and non-mask area, and fused using different fusion methods to obtain the fused non-main basis; finally, the fused main basis and the fused non-main basis are used for tensor reconstruction, and the artifact-free high dynamic range light field is obtained by angle recovery.

[0078] The present invention proposes a dynamic multi-exposure light field image fusion method based on structural consistency detection, and its overall implementation flow chart is as follows: Figure 1 As shown, it includes the following steps:

[0079] Step 1: Obtain M light field images with different exposure levels taken at the same spatial position, and construct a dynamic multi-exposure light field image sequence from the M light field images with different exposure levels, denoted as {L i ∈R U×V×H×W×C |1≤i≤M}; where M>1, in this embodiment, M=3, L i represents the light field image of the i-th exposure level, also the i-th light field image, R represents a real number set, L i The spatial resolution is W×H, the angular resolution is V×U, and the number of color channels is C. In this embodiment, W×H is 600×400, V×U is 9×9, and C is 3 (RGB three color channels), 1≤i≤M.

[0080] Step 2: i ∈R U×V×H×W×C The sub-aperture images of each light field image in |1≤i≤M} are aligned to the center view to obtain the corresponding registered light field image. i The corresponding registered light field image is denoted as L A(i) Among them, L A(i) ∈R U ×V×H×W×C , L A(i) The spatial resolution is W×H, the angular resolution is V×U, and the number of color channels is C.

[0081] In this embodiment, the specific process of step 2 is: using the optical flow estimation method to calculate {L i ∈R U×V×H×W×C The disparity between the central sub-aperture image and each non-central sub-aperture image of each light field image in |1≤i≤M} is calculated. Then, using reverse rendering, each non-central sub-aperture image of each light field image is mapped onto the central sub-aperture image according to the corresponding disparity relationship to achieve central view alignment, thereby obtaining the corresponding registered light field image. The non-central sub-aperture images are the sub-aperture images other than the central sub-aperture image. Here, the optical flow estimation method is specifically the SIFT flow optical flow estimation method.

[0082] Step 3: Perform high-order singular value decomposition on the angular dimension of each registered light field image to obtain the corresponding baseband tensor. The first component of the baseband tensor is used as the main basis, and the remaining basebands except the first component of the baseband tensor are used as non-main basis. A(i) The principal basis in the corresponding baseband tensor is denoted as P B(i) , the set of principal bases in the baseband tensors corresponding to all registered light field images {P B(i) |\1≤i≤M} as the multi-exposure main base sequence, and L A(i) The kth non-primary basis in the corresponding baseband tensor is denoted as P O(i),k, the set of the kth non-primary basis at the same position in the basis band tensor corresponding to all registered light field images {P O(i),k |\1≤i≤M} is used as the k-th multi-exposure non-primary basis sequence, and there are M non-primary basis sequences in the k-th multi-exposure non-primary basis sequence; wherein, 1≤k≤K, K represents the number of basebands other than the first component in the baseband tensor corresponding to each registered light field image, and K=V×U-1.

[0083] In this embodiment, in step 3, P B(i) and P O(i),k The acquisition process is:

[0084] Step 3.1: L A(i) Perform high-order singular value decomposition, the decomposition formula is: L A(i) =S i ×1O (1) ×2O (2) ×3O (3) ×4O (4) ×5O (5) ; Among them, S i Indicates L A(i) The core tensor obtained after high-order singular value decomposition, S i ∈R U'×V'×H'×W'×C' , U' and V', H', W', C' are S i The five dimensions of O (1) and O (2) is the orthogonal factor matrix of two angle dimensions, O (1) ∈R U×U' , O (2) ∈R V×V' , O (3) and O (4) is the orthogonal factor matrix of two spatial dimensions, O (3) ∈R H×H' , O (4) ∈R W×W' , O (5) Represents the orthogonal factor matrix of the color channel dimension, O (5) ∈R C×C' .

[0085] Step 3.2: Place L A(i) =S i ×1O (1) ×2O (2) ×3O (3) ×4O (4) ×5O (5) Convert to The superscript "T" indicates the transpose of a vector or matrix.

[0086] Step 3.3: On the basis of , in order to achieve dimensionality reduction of light field data, it is only necessary to perform singular value decomposition on the angular dimension of the light field, that is, to calculate L A(i) The baseband tensor obtained by high-order singular value decomposition of the angular dimension is denoted as P i , P i The first component in the main basis P B(i) , P i The kth baseband except the first component is taken as the kth non-primary basis P O(i),k Here, P i It contains all baseband information. The main basis contains consistent spatial information under all viewing angles. The remaining basebands except the first component, namely the non-main basis, mainly capture the angular high-frequency information of different sub-aperture images.

[0087] Step 4: Input the multi-exposure main base sequence into the main base fusion module based on structure consistency detection to obtain the fused main base, denoted as P BB ; Among them, after obtaining the fusion main base P BB In the process of obtaining the structural consistency detection binary image of each main base in the multi-exposure main base sequence.

[0088] In this embodiment, if Figure 2 As shown, in step 4, the main basis fusion module based on structure consistency detection is used to process the multi-exposure main basis sequence to obtain the fused main basis P BB The process is:

[0089] Step 4.1: Segment each main basis in the multi-exposure main basis sequence into multiple overlapping color blocks in RGB space. When segmenting the overlapping color blocks, a sliding window is used to slide in the main basis with a step size of 2 pixels to achieve segmentation. The size of the color block is 24×24.

[0090] Step 4.2: Select a reference master basis from the multi-exposure master basis sequence and record the reference master basis as P r ; Then use the intensity mapping function and the reference main basis P r To generate a multi-exposure virtual main base sequence, the i-th virtual main base in the multi-exposure virtual main base sequence is recorded as P X(i) ; Then traverse each main basis in the multi-exposure main basis sequence and define the currently traversed main basis as the current main basis; wherein the reference main basis is {L i ∈R U×V×H×W×C The principal basis in the baseband tensor corresponding to the registered light field image corresponding to the light field image with the best exposure level in |1≤i≤M} is selected. In this embodiment, an underexposed light field image, a medium-exposed light field image, and an overexposed light field image are captured at the same spatial position, and the principal basis corresponding to the medium-exposed light field image is selected as the reference principal basis.

[0091] Step 4.3: Set the current main base to P B(i) , P B(i) The jth color block in is denoted as x i,j , and x i,j Expressed as the signal strength component c i,j , signal structure component s i,j and the average intensity component l i,j Three parts, Among them, when segmenting overlapping color blocks, a sliding window is used with a step size of 2 pixels in P B(i) Segmentation is achieved by sliding in the middle. The size of the color block is 24×24, 1≤j≤J, J represents P B(i) The total number of colored blocks in c i,j and l i,j Both are scalars, s i,j is a vector, and “||||” is the L2 norm calculation symbol. Represents x i,j The average value of all pixel values in , "·" is the dot multiplication operator,

[0092] Step 4.4: P B(i) Partition and set P B(i) Divided into overexposed area and non-overexposed area; then according to P B(i) The over-exposure area and the non-over-exposure area are obtained. B(i) The corresponding over-exposed binary image is recorded as

[0093] In this embodiment, the specific process of step 4.4 is:

[0094] Step 4.4.1: Place P B(i) The color space is converted from RGB space to HSV model; then P is converted to HSV model under HSV model. B(i) Divide into multiple overlapping blocks; then compare P B(i) The size of the region division index of each overlapping block in the image is compared with the set division threshold. If the region division index is greater than or equal to the set division threshold, the overlapping block is considered to belong to the overexposed area; if the region division index is less than the set division threshold, the overlapping block is considered to belong to the non-overexposed area. When segmenting overlapping blocks, a sliding window is used with a step size of 2 pixels in P B(i) Segmentation is achieved by sliding in the middle. The size of the overlapping block is 24×24. The region division index is equal to the number of pixels in the overlapping block with a V component value greater than 240 divided by the total number of pixels contained in the overlapping block. The set division threshold is equal to the product of 0.8 and the total number of pixels contained in the overlapping block.

[0095] Step 4.4.2: Let Indicates PB(i) The corresponding over-exposed binary image, The number of pixels in each row is the same as P B(i) The number of overlapping blocks in each row is equal, The number of pixels in each column is the same as P B(i) The number of overlapping blocks in each column is equal. If P B(i) One of the overlapping blocks belongs to the overexposed area, then The pixel value of the corresponding pixel in the B(i) One of the overlapping blocks belongs to the non-overexposed area, then The pixel value of the pixel at the corresponding position in is 0.

[0096] Step 4.5: Use the perceptual hashing algorithm to obtain P B(i) The corresponding first structural consistency region detection binary image is recorded as The acquisition process is: calculate the reference main basis P r Each colored block in P B(i) The Hamming distance of the hash value of the corresponding color block in the reference base P r The j-th color block x in r,j With P B(i) The j-th color block x in i,j The Hamming distance of the hash value is recorded as δ i,j , δ i,j =F PHA (x r,j ,x i,j ); Then according to the reference main base P r All the colored blocks in P B(i) The Hamming distance of the hash value of the corresponding color block in is obtained The number of pixels in each row is the same as P B(i) The number of colored blocks in each row is equal. The number of pixels in each column is the same as P B(i) The number of color blocks in each column is equal. If the reference base P r A colored block in P B(i) The Hamming distance of the hash value of the corresponding color block in is less than or equal to the threshold th1, then The pixel value of the corresponding pixel point in the reference position is 0; if the reference base P r A colored block in P B(i) The Hamming distance of the hash value of the corresponding color block in is greater than the threshold th1, then The pixel value of the corresponding pixel point in the P is 1; where 1≤j≤J, J represents P B(i) The total number of colored blocks in δ i,j ∈[0,64],FPHA () indicates that the perceptual hash algorithm operation is performed, and th1=5.

[0097] Here, the perceptual hash algorithm is an existing algorithm, which is used to calculate δ i,j The general process is: P r and P B(i) The size of the jth color block in each is reduced to 8×8 and then converted into a grayscale block; then the DCT (cosine discrete transform) coefficient of each pixel in each grayscale block is calculated to obtain the DCT coefficient matrix; then the mean of the DCT coefficient matrix of each grayscale block is calculated; and then the P r The mean and P of the DCT coefficient matrix of the grayscale block corresponding to the j-th color block B(i) Based on the mean value of the DCT coefficient matrix of the grayscale block corresponding to the j-th color block, δ i,j .

[0098] Step 4.6: Use the normalized cross-correlation algorithm to obtain P B(i) The corresponding second structure consistency region detection binary image is recorded as The acquisition process is: calculate the reference main basis P r The signal structure component of each color block in P B(i) The inner product of the signal structure component of the corresponding color block in the reference base P r The j-th color block x in r,j The signal structure component s r,j With P B(i) The j-th color block x in i,j The signal structure component s i,j The inner product is denoted as θ i,j , Then according to the reference main base P r The signal structure components of all color blocks in P B(i) The inner product of the signal structure component of the corresponding color block in The number of pixels in each row is the same as P B(i) The number of colored blocks in each row is equal. The number of pixels in each column is the same as P B(i) The number of color blocks in each column is equal. If the reference base P r The signal structure component of a color block in P B(i) The inner product of the signal structure component of the corresponding color block is greater than or equal to the threshold constant th2, then The pixel value of the corresponding pixel point in the reference position is 0; if the reference base P r The signal structure component of a color block in P B(i)The inner product of the signal structure component of the corresponding color block is less than the threshold constant th2, then The pixel value of the corresponding pixel point in θ is 1; i,j ∈[-1,1],l r,j Indicates the reference primary base P r The j-th color block x in r,j The average intensity component of , ε is an infinitesimal number used to prevent the denominator from being 0, th2 = 0.8.

[0099] Step 4.7: Calculate the reference principal basis P r The average intensity component of each color block in P X(i) The average intensity difference between the average intensity components of the corresponding color blocks in the reference base P r The j-th color block x in r,j The average intensity component l r,j With P X(i) The j-th color block x in X(i),j The average intensity component l X(i),j The average intensity difference between i,j ,ζ i,j =||l r,j -l X(i),j ||; then according to the reference main base P r The average intensity component of all color blocks in P X(i) The average intensity difference between the average intensity components of the corresponding color blocks in The number of pixels in each row is the same as P B(i) The number of colored blocks in each row is equal. The number of pixels in each column is the same as P B(i) The number of color blocks in each column is equal. If the reference base P r The average intensity component of a color block in P X(i) The average intensity difference between the average intensity components of the corresponding color blocks in is greater than or equal to the threshold constant th3, then The pixel value of the corresponding pixel point in the reference position is 0; if the reference base P r The average intensity component of a color block in P X(i) The average intensity difference between the average intensity components of the corresponding color blocks in is less than the threshold constant th3, then The pixel value of the corresponding pixel point in the X(i),j Obtained according to the process of step 4.3, th3 = 0.1.

[0100] Step 4.8: According to Get P B(i)The structural consistency detection binary map is denoted as Z i ,

[0101] Step 4.9: Determine P B(i) Whether each colored block in P belongs to the structural consistency area, B(i) The j-th color block x in i,j , if Z i Zhong and x i,j The pixel value of the pixel at the corresponding position is 0, then x i,j Belongs to the structural consistency region; if Z i Zhong and x i,j The pixel value of the pixel at the corresponding position is 1, then x i,j It belongs to the area of structural inconsistency.

[0102] Step 4.10: For P B(i) The colored blocks in the structural inconsistency area are used to X(i) The corresponding colored blocks in are replaced to achieve compensation, that is: assuming x i,j Belong to the structural inconsistency area, then use P X(i) The j-th color block x in X(i),j Replace x i,j ; will be P B(i) The compensation main basis obtained after compensating all the color blocks belonging to the structural inconsistency area is recorded as P' B(i) .

[0103] Step 4.11: Take the next traversed main basis in the multi-exposure main basis sequence as the current main basis, and then return to step 4.3 to continue executing until the compensated main basis corresponding to each main basis in the multi-exposure main basis sequence is obtained.

[0104] Step 4.12: Based on the color blocks at the same position in the compensation main basis corresponding to all the main basis in the multi-exposure main basis sequence, that is, a total of M color blocks, a corresponding fused image block is reconstructed. i,j The fused image block reconstructed from the M color blocks of |1≤i≤M} is recorded as Expressed as signal strength component Signal structure component and the average intensity component Three parts, Among them, x' i,j Represents P' B(i) The jth color block in and are scalars, is a vector, max() is the maximum value function. express The average signal structure component of ω1() is a weighted function, and ω1() is expressed by an exponential function. β is the exponential parameter, β = 4, μ' i Represents P' B(i) The global mean of ω2() is the weighting function, and ω2() adopts a two-dimensional normal distribution. exp() represents the exponential function with the natural base e as the base, e=2.71…,μ c 、l c , σ x , σ y are the parameters of the two-dimensional normal distribution, σ x =0.2,σ y =0.5, μ c =0.5, l c =0.5, c' i,j represents x' i,j The signal strength component, s' i,j represents x' i,j The signal structure component, l' i,j represents x' i,j The average intensity component of represents x' i,j The average value of all pixel values in .

[0105] Step 4.13: Press P B(i) The reverse process of segmenting into overlapping colored blocks is to All fused image blocks are recombined into a main basis as the fusion main basis P BB ; Among them, the average value of the pixel values of the pixels at the same position in the overlapping area during reorganization is used as the final pixel value after reorganization.

[0106] Step 5: Input the k-th multi-exposure non-primary base sequence into the remaining baseband fusion modules based on mask partitioning to obtain the k-th fused non-primary base, denoted as P OB,k ; Among them, when dividing the mask, the structural consistency detection binary image corresponding to the main basis is used as the mask.

[0107] In this embodiment, if Figure 2 As shown, in step 5, the remaining baseband fusion modules based on mask division are used to process the k-th multi-exposure non-primary base sequence to obtain the k-th fused non-primary base sequence P OB,k The process is:

[0108] Step 5.1: Segment each non-primary basis in the k-th multi-exposure non-primary basis sequence into multiple overlapping color blocks in RGB space. Segment the overlapping color blocks using a sliding window with a step size of 2 pixels across the non-primary basis. The size of the color blocks is 24×24.

[0109] Step 5.2: Classify all color blocks in each non-main base in the k-th multi-exposure non-main base sequence into mask areas and non-mask areas. O(i),k , Z i As a mask to P O(i),k The process of classifying each color block in P is as follows: O(i),k The j-th color block x in i,k,j , if Z i Zhong and x i,k,j The pixel value of the pixel at the same position is 1, then x i,k,j Belongs to the mask area; if Z i Zhong and x i,k,j The pixel value of the pixel at the same position is 0, so it is considered that x i,k,j Belongs to the non-mask area.

[0110] Step 5.3: Get the kth fused non-primary basis P OB,k The non-mask area is denoted as P Os,k , Among them, max() is the maximum value function, and "·" is the dot multiplication operator. Indicates Z i The complementary graph obtained by performing the inverse operation.

[0111] Step 5.4: Get the kth fused non-primary basis P OB,k The mask area is denoted as P Od,k , the acquisition process is:

[0112] Step 5.4.1: For each color block in the mask area in the non-primary basis in the k-th multi-exposure non-primary basis sequence, use P X(i) The corresponding color blocks in P are replaced to achieve compensation, that is: O(i),k , assuming x i,k,j Belongs to the mask area, then use P X(i) The j-th color block x in X(i),j Replace x i,k,j ; will be P O(i),k The compensated non-primary basis obtained after compensating all the color blocks belonging to the mask area is recorded as P' O(i),k .

[0113] Step 5.4.2: Reconstruct a corresponding fused image block based on the color blocks at the same position in the compensated non-main base corresponding to all non-main bases in the k-th multi-exposure non-main base sequence, that is, a total of M color blocks. i,k,j The fused image block reconstructed from the M color blocks of |1≤i≤M} is recorded as Expressed as signal strength component Signal structure component and the average intensity component Three parts, Among them, x' i,k,j Represents P' O(i),k The jth color block in Obtained as in step 4.12.

[0114] Step 5.4.3: Press P O(i),k The reverse process of segmenting into overlapping colored blocks is to All fused image blocks are reorganized into a mask region P Od,k ; Among them, the average value of the pixel values of the pixels at the same position in the overlapping area during reorganization is used as the final pixel value after reorganization.

[0115] Step 5.5: Get the kth fused non-primary basis P OB,k , P OB,k =P Os,k +P Od,k .

[0116] Step 6: According to P BB and the set of all fused non-primary bases {P OB,k |1≤k≤K}, and the fused baseband tensor is obtained, which is recorded as Then combine The orthogonal factor matrices of the two angular dimensions when the angular dimension of the registered light field image is decomposed by high-order singular value are reconstructed to restore the light field structure. The obtained fused light field image is recorded as

[0117] Step 7: Right The angle information of the image is restored to obtain a fused light field image containing complete angle information, which is recorded as L H .

[0118] In this embodiment, the specific process of step 7 is: according to {L i ∈R U×V×H×W×C The disparity between the central sub-aperture image of the light field image with the best exposure in |1≤i≤M} and each non-central sub-aperture image is converted into The non-central sub-aperture images aligned with the central views of the image are mapped to their corresponding positions to restore the angle information, and L H .

[0119] In order to further illustrate the feasibility and effectiveness of the method of the present invention, an experiment was conducted to verify the method of the present invention.

[0120] To validate the performance of our method, we used a dataset of multi-exposure light field images of dynamic scenes as a test dataset. This dataset includes 27 sequences of dynamic multi-exposure light field images with exposure values ranging from -2EV to +2EV. Each scene contains three light field images with different exposure levels, each differing by 2EV. These scenes include both indoor and outdoor scenes, rigid and dynamic objects, the motion of near and far objects, large and small motions, and the motion of both single and multiple people.

[0121] In order to quantitatively evaluate the performance of the proposed method, the proposed method is compared with existing two-dimensional image multi-exposure fusion methods, namely, a simple and practical alternative high dynamic range photography method proposed by Mertens et al., an image fusion method based on guided filtering (GFF) proposed by Li et al., a ghost-free multi-exposure fusion method using dense SIFT (Scale-Invariant Feature Transform) (DSIFT-MEF) proposed by Liu et al., a ghost-free multi-exposure image fusion method based on dense SIFT descriptor and guided filtering proposed by Hayat et al., a fast multi-scale multi-exposure fusion method based on structured block decomposition (FMSSPD-MEF) proposed by Li et al., and a multi-exposure fusion method with detail-preserving and edge-preserving structured block decomposition (MESPD) proposed by Li et al.

[0122] Here, considering that there is no ground-truth image for the multi-exposure light field image dataset as a reference image; at the same time, the dataset contains multi-exposure light field image sequences of dynamic scenes, most objective evaluation indicators of multi-exposure image fusion methods are not applicable here. Therefore, a quality model MEF-SSIM for multi-exposure image fusion (MEF) suitable for dynamic scenes is adopted. d As an objective evaluation index, it is used to evaluate the performance of different MEF methods in obtaining high dynamic range light fields. dIt is a quality evaluation index based on the principle of structural similarity, which takes into account the influence of static and dynamic regions on the fusion quality. This quality evaluation index measures the structural similarity between the corresponding sequences in two regions of the image respectively, and combines the quality measurement values of the two regions into an overall quality score. The larger the value, the better the performance of the method. Table 1 shows the MEF-SSIM of different MEF methods on the multi-exposure light field image dataset. d Scores, where bold indicates the best performing method and underlined indicates the second best performing method.

[0123] Table 1 Comparison of the performance of the method of the present invention and the existing two-dimensional image multi-exposure fusion method in MEF-SSIM d Comparison of objective evaluation indicators

[0124] sequence Mertens GFF DSIFT-MEF Hayat19 FMSSPD-MEF MESPD Method of the present invention 1 0.9058 <![CDATA[ 0.9262 ]]> 0.9050 0.9339 0.9180 0.9222 0.8996 2 0.8127 0.7883 0.8166 <![CDATA[ 0.8260 ]]> 0.8048 0.8142 0.9010 3 0.8920 0.8874 0.8655 0.8945 0.9310 <![CDATA[ 0.9289 ]]> 0.8453 4 0.7907 0.7396 0.7964 0.7779 0.8421 <![CDATA[ 0.8440 ]]> 0.8594 5 0.7736 0.7610 0.8766 0.8337 0.8494 0.8469 <![CDATA[ 0.8676 ]]> 6 0.9079 0.8885 0.8775 0.8810 <![CDATA[ 0.9225 ]]> 0.9264 0.8331 7 0.8486 0.8501 0.8638 0.8449 <![CDATA[ 0.8920 ]]> 0.8929 0.8868 8 0.7893 0.7014 0.7782 0.7987 0.8091 <![CDATA[ 0.8157 ]]> 0.9270 9 0.7952 0.7850 0.8034 0.8012 0.8173 <![CDATA[ 0.8204 ]]> 0.8837 10 0.8866 0.8765 0.8768 0.8953 <![CDATA[ 0.9120 ]]> 0.9151 0.8679 11 0.8793 0.8839 0.8044 0.8723 <![CDATA[ 0.9144 ]]> 0.9178 0.8424 12 0.8293 0.8168 0.8424 0.8404 <![CDATA[ 0.8477 ]]> 0.8524 0.8017 13 0.7780 0.7737 0.7927 <![CDATA[ 0.8759 ]]> 0.8218 0.8251 0.8774 14 0.8404 0.8354 0.8511 <![CDATA[ 0.8974 ]]> 0.8764 0.8784 0.9021 15 0.8322 0.8281 0.7926 0.7752 0.8543 <![CDATA[ 0.8574 ]]> 0.8833 16 0.8036 0.8010 0.7785 0.8038 0.8424 <![CDATA[ 0.8483 ]]> 0.8768 17 0.8743 0.8415 0.8725 0.8687 <![CDATA[ 0.8869 ]]> 0.8852 0.8887 18 0.8850 0.8830 0.8856 0.8378 <![CDATA[ 0.9082 ]]> 0.9128 0.8836 19 0.8261 0.8263 0.7901 0.8124 0.8677 <![CDATA[ 0.8711 ]]> 0.8856 20 0.8219 0.7754 0.8505 <![CDATA[ 0.8924 ]]> 0.8372 0.8370 0.9178 21 0.8723 0.8241 0.8148 0.7697 <![CDATA[ 0.8937 ]]> 0.8970 0.8378 22 0.8425 0.8550 0.8862 <![CDATA[ 0.9000 ]]> 0.8741 0.8850 0.9091 23 0.9125 0.8999 0.9204 0.9127 <![CDATA[ 0.9300 ]]> 0.9332 0.9052 24 0.8415 0.8498 0.8431 0.8314 0.8727 <![CDATA[ 0.8778 ]]> 0.9036 25 0.8306 0.8494 <![CDATA[ 0.8762 ]]> 0.8278 0.8640 0.8731 0.9047 26 0.8036 0.8079 0.8275 0.8118 0.8348 <![CDATA[ 0.8394 ]]> 0.8481 27 0.8658 0.8973 0.8434 0.8463 <![CDATA[ 0.9106 ]]> 0.9160 0.9073 average 0.8423 0.8316 0.8419 0.8468 0.8717 <![CDATA[ 0.8753 ]]> 0.8795

[0125] From the MEF-SSIM listed in Table 1 d The scores show that the proposed method demonstrates comparable performance to the FMSSPD-MEF and MESPD methods in fusion light fields across 27 scenes, and significantly outperforms Mertens' method, GFF, DSIFT-MEF, and Hayat's methods. This demonstrates that, compared to the aforementioned existing 2D image multi-exposure fusion methods, the proposed method can effectively combine low-dynamic-range images from underexposed, midexposed, and overexposed light fields, effectively improving the dynamic range of light field images and producing high-quality, artifact-free, high-dynamic-range light fields.

[0126] Figure 3a 、 Figure 3b 、 Figure 3c The central sub-aperture images of the under-exposed light field image, the medium-exposed light field image, and the over-exposed light field image in the fourth test light field image sequence are given respectively. Figures 3d to 3j The central sub-aperture images of the fused light field images obtained by processing the fourth test light field image sequence using the method of Mertens et al., the method of Li et al. (GFF), the method of Liu et al. (DSIFT-MEF), the method of Hayat et al., the method of Li et al. (FMSSPD-MEF), the method of Li et al. (MESPD) and the method of the present invention are given respectively. Figure 3a 、 Figure 3b 、 Figure 3c ,from Figures 3d to 3iIt can be seen from the figure that the method of Mertens et al. and the method of Li et al. (GFF) cannot obtain good fusion results in dynamic scenes. There are large artifacts in the doll in the front and there is a certain color distortion in the glass area of the background. The method of Liu et al. (DSIFT-MEF) can solve the color distortion problem of the above methods. The artifact phenomenon is alleviated to a certain extent through SIFT descriptor matching, but it cannot completely solve the problem for objects with large motion amplitudes. The method of Li et al. (FMSSPD-MEF) and the method of Li et al. (MESPD) enhance the content information and color of the fusion area. Figure 3h and Figure 3i As can be seen from the local magnified image in the figure, it cannot completely solve the artifact problem; Hayat et al.'s method can detect the moving area well, but it cannot calculate the weight map for the fusion of light spots and dolls. Figure 3j As can be seen from the figure, the proposed method is able to effectively remove artifacts in moving areas. The lines representing the EPI magnification images show some distortion in the EPI oblique lines of the Mertens method, the GFF method, and the DSIFT-MEF method. The color information of the Hayat method and the FMSSPD-MEF method is not rich enough. However, the MESPD method and the proposed method are able to better maintain angular consistency and color details.

[0127] In addition to comparing the performance of the fused light field images obtained by observing the fusion results of the central sub-aperture image and the straight line preservation of the EPI, in order to further visualize the imaging quality of the fused light field obtained by different methods, the light field image depth estimation algorithm proposed by Chen et al. is used to measure the ability of different methods to preserve the geometric characteristics of the light field. The more accurate the depth map obtained, the better the angle consistency of the method is maintained. Figure 4a 、 Figure 4b 、 Figure 4c The central sub-aperture images of the under-exposed light field image, medium-exposed light field image, and over-exposed light field image in the 22nd test light field image sequence are given respectively. Figure 4d to Figure 4m The depth maps of the fused light field image obtained by processing the 22nd test light field image sequence using the method of Mertens et al., the method of Li et al. (GFF), the method of Liu et al. (DSIFT-MEF), the method of Hayat et al., the method of Li et al. (FMSSPD-MEF), the method of Li et al. (MESPD) and the method of the present invention are given respectively. Figure 4a 、 Figure 4b 、 Figure 4c The depth maps shown cannot show a relatively complete scene object, such as Figure 4d 、 Figure 4e 、 Figure 4fThe lamp and desk area in the book section. Figure 4g 、 4h , 4i, 4j, 4k, Figure 4l 、 4m As can be seen from the depth maps shown, the depth maps of the fused light field images obtained by all multi-exposure image fusion methods are significantly improved compared to the depth maps obtained by the original multi-exposure light field image sequence. Figure 4m It can be seen from the figure that the method of the present invention can better complete the depth estimation of the image in the wall area and the book part.

[0128] In summary, the method of the present invention can better maintain the geometric structure and angular consistency of the light field, and thus optimize the generation of the depth map of the fused light field image.

Claims

1. A dynamic multi-exposure light field image fusion method based on structural consistency detection, characterized by The following steps are involved: Step 1: Obtain M light field images with different exposure levels taken at the same spatial position, and construct a dynamic multi-exposure light field image sequence from the M light field images with different exposure levels, denoted as {L i ∈R U×V×H×W×C |1≤i≤M}; where M>1, L i represents the light field image of the i-th exposure level, also the i-th light field image, R represents a real number set, L i The spatial resolution is W×H, the angular resolution is V×U, the number of color channels is C, 1≤i≤M; Step 2: i ∈R U×V×H×W×C The sub-aperture images of each light field image in |1≤i≤M} are aligned to obtain the corresponding registered light field image. i The corresponding registered light field image is denoted as L A(i) Among them, L A(i) ∈R U×V×H×W×C , L A(i) The spatial resolution is W×H, the angular resolution is V×U, and the number of color channels is C; Step 3: Perform high-order singular value decomposition on the angular dimension of each registered light field image to obtain the corresponding baseband tensor. The first component of the baseband tensor is used as the main basis, and the remaining basebands except the first component of the baseband tensor are used as non-main basis. A(i) The principal basis in the corresponding baseband tensor is denoted as P B(i) , the set of principal bases in the baseband tensors corresponding to all registered light field images {P B(i) |1≤i≤M} as the multi-exposure main base sequence, and L A(i) The kth non-primary basis in the corresponding baseband tensor is denoted as P O(i),k , the set of the kth non-primary basis at the same position in the basis band tensor corresponding to all registered light field images {P O(i),k |1≤i≤M} is used as the k-th multi-exposure non-primary basis sequence, and there are M non-primary basis in the k-th multi-exposure non-primary basis sequence; where 1≤k≤K, K represents the number of basebands except the first component in the baseband tensor corresponding to each registered light field image, K=V×U-1; Step 4: Input the multi-exposure main base sequence into the main base fusion module based on structure consistency detection to obtain the fused main base, denoted as P BB ; Among them, after obtaining the fusion main base P BB In the process of obtaining the structural consistency detection binary image of each main base in the multi-exposure main base sequence; Step 5: Input the k-th multi-exposure non-primary base sequence into the remaining baseband fusion modules based on mask partitioning to obtain the k-th fused non-primary base, denoted as P OB,k ; Among them, when dividing the mask, the structural consistency detection binary image corresponding to the main basis is used as the mask; Step 6: According to P BB and the set of all fused non-primary bases {P OB,k |1≤k≤K}, and the fused baseband tensor is obtained, which is recorded as Then combine The orthogonal factor matrices of the two angular dimensions when the angular dimension of the registered light field image is decomposed by high-order singular value are reconstructed to restore the light field structure. The obtained fused light field image is recorded as Step 7: Right The angle information of the image is restored to obtain a fused light field image containing complete angle information, which is recorded as L H .

2. The dynamic multi-exposure light field image fusion method based on structural consistency detection according to claim 1, characterized in that The specific process of step 2 is: using the optical flow estimation method to calculate {L i ∈R U×V×H×W×C The disparity between the central subaperture image and each non-central subaperture image of each light field image in |1≤i≤M}; Then, reverse rendering is used to map the non-central sub-aperture images of each light field image onto the central sub-aperture image according to the corresponding parallax relationship to achieve central view alignment and obtain the corresponding registered light field image; among which, the non-central sub-aperture images are the remaining sub-aperture images except the central sub-aperture image.

3. The dynamic multi-exposure light field image fusion method based on structural consistency detection according to claim 2, characterized in that The optical flow estimation method is specifically a SIFT flow optical flow estimation method.

4. The dynamic multi-exposure light field image fusion method based on structural consistency detection according to any one of claims 1 to 3, characterized in that In step 3, P B(i) and P O(i),k The acquisition process is: Step 3.1: L A(i) Perform high-order singular value decomposition, the decomposition formula is: L A(i) =S i ×1O (1) ×2O (2) ×3O (3) ×4O (4) ×5O (5) ; Among them, S i Indicates L A(i) The core tensor obtained after high-order singular value decomposition, S i ∈R U'×V'×H'×W'×C' , U' and V', H', W', C' are S i The five dimensions of O (1) and O (2) is the orthogonal factor matrix of two angle dimensions, O (1) ∈R U×U' , O (2) ∈R V×V' , O (3) and O (4) is the orthogonal factor matrix of two spatial dimensions, O (3) ∈R H×H' , O (4) ∈R W×W' , O (5) Represents the orthogonal factor matrix of the color channel dimension, O (5) ∈R C×C' ; Step 3.2: Place L A(i) =S i ×1O (1) ×2O (2) ×3O (3) ×4O (4) ×5O (5) Convert to Where, the superscript "T" indicates the transpose of a vector or matrix; Step 3.3: Based on the calculation of L A(i) The baseband tensor obtained by high-order singular value decomposition of the angular dimension is denoted as P i , P i The first component in the main basis P B(i) , P i The kth baseband except the first component is taken as the kth non-primary basis P O(i),k .

5. The dynamic multi-exposure light field image fusion method based on structural consistency detection according to claim 4, characterized in that In step 4, the main base fusion module based on structure consistency detection is used to process the multi-exposure main base sequence to obtain the fused main base P BB The process is: Step 4.1: Segment each primary basis in the multi-exposure primary basis sequence into multiple overlapping color blocks in RGB space. Segment the overlapping color blocks using a sliding window with a step size of 2 pixels across the primary basis. The size of the color blocks is 24×24. Step 4.2: Select a reference master basis from the multi-exposure master basis sequence and record the reference master basis as P r ; Then use the intensity mapping function and the reference main basis P r To generate a multi-exposure virtual main base sequence, the i-th virtual main base in the multi-exposure virtual main base sequence is recorded as P X(i) ; Then traverse each main basis in the multi-exposure main basis sequence and define the currently traversed main basis as the current main basis; wherein the reference main basis is {L i ∈R U×V×H×W×C The principal basis in the baseband tensor corresponding to the registered light field image corresponding to the light field image with the best exposure in |1≤i≤M}; Step 4.3: Set the current main base to P B(i) , P B(i) The jth color block in is denoted as x i,j , and x i,j Expressed as signal strength component c i,j , signal structure component s i,j and the average intensity component l i,j Three parts, Among them, when segmenting overlapping color blocks, a sliding window is used with a step size of 2 pixels in P B(i) Segmentation is achieved by sliding in the middle. The size of the color block is 24×24, 1≤j≤J, J represents P B(i) The total number of colored blocks in c i,j and l i,j Both are scalars, s i,j is a vector, "|| ||" is the L2 norm calculation symbol, Represents x i,j The average value of all pixel values in , "·" is the dot multiplication operator, Step 4.4: P B(i) Partition and set P B(i) Divided into overexposed area and non-overexposed area; then according to P B(i) The over-exposure area and the non-over-exposure area are obtained. B(i) The corresponding over-exposed binary image is recorded as Step 4.5: Use the perceptual hashing algorithm to obtain P B(i) The corresponding first structural consistency region detection binary image is recorded as The acquisition process is: calculate the reference main basis P r Each colored block in P B(i) The Hamming distance of the hash value of the corresponding color block in the reference base P r The j-th color block x in r,j With P B(i) The j-th color block x in i,j The Hamming distance of the hash value is recorded as δ i,j , δ i,j =F PHA (x r,j ,x i,j ); Then according to the reference main base P r All the colored blocks in P B(i) The Hamming distance of the hash value of the corresponding color block in is obtained The number of pixels in each row is the same as P B(i) The number of colored blocks in each row is equal. The number of pixels in each column is the same as P B(i) The number of color blocks in each column is equal. If the reference base P r A colored block in P B(i) The Hamming distance of the hash value of the corresponding color block in is less than or equal to the threshold th1, then The pixel value of the corresponding pixel point in the reference position is 0; if the reference base P r A colored block in P B(i) The Hamming distance of the hash value of the corresponding color block in is greater than the threshold th1, then The pixel value of the corresponding pixel point in the P is 1; where 1≤j≤J, J represents P B(i) The total number of colored blocks in δ i,j ∈[0,64],F PHA () indicates the perceptual hash algorithm operation, th1 = 5; Step 4.6: Use the normalized cross-correlation algorithm to obtain P B(i) The corresponding second structure consistency region detection binary image is recorded as The acquisition process is: calculate the reference main basis P r The signal structure component of each color block in P B(i) The inner product of the signal structure component of the corresponding color block in the reference base P r The j-th color block x in r,j The signal structure component s r,j With P B(i) The j-th color block x in i,j The signal structure component s i,j The inner product is denoted as θ i,j , Then according to the reference main base P r The signal structure components of all color blocks in P B(i) The inner product of the signal structure component of the corresponding color block in is obtained The number of pixels in each row is the same as P B(i) The number of colored blocks in each row is equal. The number of pixels in each column is the same as P B(i) The number of color blocks in each column is equal. If the reference base P r The signal structure component of a color block in P B(i) The inner product of the signal structure component of the corresponding color block is greater than or equal to the threshold constant th2, then The pixel value of the corresponding pixel point in the reference position is 0; if the reference base P r The signal structure component of a color block in P B(i) The inner product of the signal structure component of the corresponding color block is less than the threshold constant th2, then The pixel value of the corresponding pixel point in θ is 1; i,j ∈[-1,1],l r,j Indicates the reference primary base P r The j-th color block x in r,j The average intensity component, ε is an infinitesimal number used to prevent the denominator from being 0, th2 = 0.8; Step 4.7: Calculate the reference principal basis P r The average intensity component of each color block in P X(i) The average intensity difference between the average intensity components of the corresponding color blocks in the reference base P r The j-th color block x in r,j The average intensity component l r,j With P X(i) The j-th color block x in X(i),j The average intensity component l X(i),j The average intensity difference between i,j ,ζ i,j =||l r,j -l X(i),j ||; then according to the reference main base P r The average intensity component of all color blocks in P X(i) The average intensity difference between the average intensity components of the corresponding color blocks in The number of pixels in each row is the same as P B(i) The number of colored blocks in each row is equal. The number of pixels in each column is the same as P B(i) The number of color blocks in each column is equal. If the reference base P r The average intensity component of a color block in P X(i) The average intensity difference between the average intensity components of the corresponding color blocks in is greater than or equal to the threshold constant th3, then The pixel value of the corresponding pixel point in the reference position is 0; if the reference base P r The average intensity component of a color block in P X(i) The average intensity difference between the average intensity components of the corresponding color blocks in is less than the threshold constant th3, then The pixel value of the pixel at the corresponding position in is 1; where th3 = 0.1; Step 4.8: According to Get P B(i) The structural consistency detection binary map is denoted as Z i , Step 4.9: Determine P B(i) Whether each colored block in P belongs to the structural consistency area, B(i) The j-th color block x in i,j , if Z i Zhong and x i,j The pixel value of the pixel at the corresponding position is 0, then x i,j Belongs to the structural consistency region; if Z i Zhong and x i,j The pixel value of the pixel at the corresponding position is 1, then x i,j It belongs to the area of structural inconsistency; Step 4.10: For P B(i) The colored blocks in the structural inconsistency area are used to X(i) The corresponding colored blocks in are replaced to achieve compensation, that is: assuming x i,j Belong to the structural inconsistency area, then use P X(i) The j-th color block x in X(i),j Replace x i,j ; will be P B(i) The compensation main basis obtained after compensating all the color blocks belonging to the structural inconsistency area is recorded as P' B(i) ; Step 4.11: Use the next traversed main basis in the multi-exposure main basis sequence as the current main basis, and then return to step 4.3 to continue executing until the compensated main basis corresponding to each main basis in the multi-exposure main basis sequence is obtained; Step 4.12: Based on the color blocks at the same position in the compensation main basis corresponding to all the main basis in the multi-exposure main basis sequence, that is, a total of M color blocks, a corresponding fused image block is reconstructed. i,j The fused image block reconstructed from the M color blocks of |1≤i≤M} is recorded as Expressed as signal strength component Signal structure component and the average intensity component Three parts, Among them, x' i,j Represents P' B(i) The jth color block in and are scalars, is a vector, max() is the maximum value function. express The average signal structure component of ω1() is a weighted function, and ω1() is expressed by an exponential function. β is the exponential parameter, β=4, μ' i Represents P' B(i) The global mean of ω2() is the weighting function, and ω2() adopts a two-dimensional normal distribution. exp() represents the exponential function with the natural base e as the base, e=2.71…,μ c 、l c , σ x , σ y are the parameters of the two-dimensional normal distribution, σ x =0.2,σ y =0.5, μ c =0.5, l c =0.5, c' i,j represents x' i,j The signal strength component, s' i,j represents x' i,j The signal structure component, l' i,j represents x' i,j The average intensity component of μ x'i,j represents x' i,j The average value of all pixel values in ; Step 4.13: Press P B(i) The reverse process of segmenting into overlapping colored blocks is to All fused image blocks are recombined into a main basis as the fusion main basis P BB ; Among them, the average value of the pixel values of the pixels at the same position in the overlapping area during reorganization is used as the final pixel value after reorganization.

6. The dynamic multi-exposure light field image fusion method based on structural consistency detection according to claim 5, characterized in that The specific process of step 4.4 is as follows: Step 4.4.1: Place P B(i) The color space is converted from RGB space to HSV model; Then, under the HSV model, P B(i) Divide into multiple overlapping blocks; then compare P B(i) The size of the region division index of each overlapping block in the image is compared with the set division threshold. If the region division index is greater than or equal to the set division threshold, the overlapping block is considered to belong to the overexposed area; if the region division index is less than the set division threshold, the overlapping block is considered to belong to the non-overexposed area. When segmenting overlapping blocks, a sliding window is used with a step size of 2 pixels in P B(i) Segmentation is achieved by sliding in the middle. The size of the overlapping block is 24×24. The region division index is equal to the number of pixels in the overlapping block with a V component value greater than 240 divided by the total number of pixels in the overlapping block. The segmentation threshold is set to be the product of 0.8 and the total number of pixels in the overlapping block. Step 4.4.2: Let Indicates P B(i) The corresponding over-exposed binary image, The number of pixels in each row is the same as P B(i) The number of overlapping blocks in each row is equal, The number of pixels in each column is the same as P B(i) The number of overlapping blocks in each column is equal. If P B(i) One of the overlapping blocks belongs to the overexposed area, then The pixel value of the corresponding pixel in the B(i) One of the overlapping blocks belongs to the non-overexposed area, then The pixel value of the pixel at the corresponding position in is 0.

7. The dynamic multi-exposure light field image fusion method based on structural consistency detection according to claim 5, characterized in that In step 5, the remaining baseband fusion modules based on mask division are used to process the k-th multi-exposure non-primary base sequence to obtain the k-th fused non-primary base sequence P OB,k The process is: Step 5.1: Segment each non-primary basis in the k-th multi-exposure non-primary basis sequence into multiple overlapping color blocks in RGB space. Segment the overlapping color blocks using a sliding window with a step size of 2 pixels across the non-primary basis. The size of the color blocks is 24×24. Step 5.2: Classify all color blocks in each non-main base in the k-th multi-exposure non-main base sequence into mask areas and non-mask areas. O(i),k , Z i As a mask to P O(i),k The process of classifying each color block in P is as follows: O(i),k The j-th color block x in i,k,j , if Z i Zhong and x i,k,j The pixel value of the pixel at the same position is 1, then x i,k,j belongs to the mask area; if Z i Zhong and x i,k,j The pixel value of the pixel at the same position is 0, so it is considered that x i,k,j Belongs to the non-mask area; Step 5.3: Get the kth fused non-primary basis P OB,k The non-mask area is denoted as P Os,k , Among them, max() is the maximum value function, "·" is the dot multiplication operator symbol, Indicates Z i The complementary graph obtained by performing the inverse operation; Step 5.4: Get the kth fused non-primary basis P OB,k The mask area is denoted as P Od,k , the acquisition process is: Step 5.4.1: For each color block in the mask area in the non-primary basis in the k-th multi-exposure non-primary basis sequence, use P X(i) The corresponding color blocks in P are replaced to achieve compensation, that is: O(i),k , assuming x i,k,j Belongs to the mask area, then use P X(i) The j-th color block x in X(i),j Replace x i,k,j ; will be P O(i),k The compensated non-primary basis obtained after compensating all the color blocks belonging to the mask area is recorded as P' O(i),k ; Step 5.4.2: Reconstruct a corresponding fused image block based on the color blocks at the same position in the compensated non-main base corresponding to all non-main bases in the k-th multi-exposure non-main base sequence, that is, a total of M color blocks. i,k,j The fused image block reconstructed from the M color blocks of |1≤i≤M} is recorded as Expressed as signal strength component Signal structure component and the average intensity component Three parts, Among them, x' i,k,j Represents P' O(i),k The jth color block in Obtained as in step 4.12; Step 5.4.3: Press P O(i),k The reverse process of segmenting into overlapping colored blocks is to All fused image blocks are reorganized into a mask region P Od,k ; Among them, the average value of the pixel values of the pixels at the same position in the overlapping area during reorganization is used as the final pixel value after reorganization; Step 5.5: Get the kth fused non-primary basis P OB,k , P OB,k =P Os,k +P Od,k .

8. The dynamic multi-exposure light field image fusion method based on structural consistency detection according to claim 4, characterized in that In step 6, 9. The dynamic multi-exposure light field image fusion method based on structural consistency detection according to claim 2, characterized in that The specific process of step 7 is as follows: i ∈R U×V×H×W×C The disparity between the central sub-aperture image of the light field image with the best exposure in |1≤i≤M} and each non-central sub-aperture image is converted into The non-central sub-aperture images aligned with the central views of the image are mapped to their corresponding positions to restore the angle information, and L H .

Images