Low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling

Abstract

The invention discloses a low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling, and the method specifically comprises the following steps: 1, selecting a binocular stereo image pair data set, carrying out the low-illumination preprocessing of an image pair in the data set, and obtaining a low-illumination binocular stereo image pair data set, dividing the preprocessed data set into a training set, a verification set and a test set; step 2, constructing a dark light stereo image enhancement network based on wavelet decoupling; step 3, training the network model constructed in the step 2 by using the training set preprocessed in the step 1 to obtain an optimal enhanced network model; and 4, testing the enhanced network model obtained in the step 3 by using the test set preprocessed in the step 1, and outputting an enhanced image. According to the invention, the problems of insufficient image brightness and poor color recovery generally existing in the existing stereo image enhancement technology are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and relates to a low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling. Background Art

[0002] Binocular stereo vision is an advanced technology that mimics the workings of the human visual system, specifically how we perceive three-dimensional space through our two eyes. Our eyes are approximately 6 centimeters apart, allowing each eye to view the world from a slightly different perspective. The brain calculates the distance and depth of objects by comparing these differences in perspective (i.e., parallax). Binocular stereo vision utilizes this principle by capturing two images of the same scene using two cameras. The slight difference between these two images provides the basis for the system to calculate depth information. Binocular vision is now being applied to an increasingly wide range of scenarios, from the earliest robotic vision to current fields such as autonomous driving and medical imaging.

[0003] In low-light environments (grayscale value <50, 0-255 range), such as at night or in low-light conditions indoors, binocular stereo vision systems face severe challenges. This environment often leads to a decline in binocular stereo image quality and difficulty in extracting disparity information, which seriously affects the performance and reliability of the binocular vision system. Specifically, low-light environments result in insufficient image brightness, making many details in the image difficult to discern, while increasing image noise, which interferes with the details in the image and makes the extraction of disparity information more difficult. In addition, reduced contrast is also a problem, which blurs the edges and textures in the image, affecting the stereo vision system's recognition of the edges of objects in the image, and thus affecting the calculation of depth information. The limited dynamic range further exacerbates this problem, resulting in the loss of details in the image, especially in low light conditions, where this range may be reduced, resulting in the inability to fully capture the details in the image. Summary of the Invention

[0004] The purpose of the present invention is to provide a low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling, which solves the problems of insufficient image brightness and poor color restoration that are common in existing stereo image enhancement technologies.

[0005] The technical solution adopted by the present invention is a low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling, which specifically includes the following steps: Step 1: Select a binocular stereo image pair dataset, perform low-light preprocessing on the image pairs in the dataset to obtain a low-light binocular stereo image pair dataset, and divide the preprocessed dataset into a training set, a validation set, and a test set; Step 2: Construct a dark-light stereo image enhancement network based on wavelet decoupling; Step 3: Use the training set preprocessed in step 1 to train the network model constructed in step 2 to obtain the optimal enhanced network model; Step 4: Test the enhanced network model obtained in step 3 using the test set preprocessed in step 1, and output the enhanced image.

[0006] The present invention is also characterized in that: The dark-light stereo image enhancement network based on wavelet decoupling constructed in step 2 includes: wavelet transform and downsampling module, illumination adjustment module, high-frequency guided stereo matching module, high-frequency information feature enhancement module and inverse wavelet transform image reconstruction module.

[0007] The specific process of step 2 is: Step 2.1: First input low-light binocular stereo images, which are the left view and the right view , first use 3×3 convolution to expand the channel of the left view image, and get , and then perform wavelet transform on the expanded left view features to obtain the low-frequency information of the left view after the first wavelet decomposition , as shown in formula (1), the right view is operated in the same way as the left view, as shown in formula (2): (1) (2) Among them, DWT represents the wavelet transform operation, Represents the high-frequency information in the horizontal direction of the left view after the first wavelet decomposition, Represents the high-frequency information in the vertical direction of the left view after the first wavelet decomposition, Represents the high-frequency information in the diagonal direction of the left view after the first wavelet decomposition, Represents the low-frequency information of the left view after the first wavelet decomposition; Represents the high-frequency information in the horizontal direction of the right view after the first wavelet decomposition, Represents the high-frequency information in the vertical direction of the right view after the first wavelet decomposition, Represents the high-frequency information in the diagonal direction of the right view after the first wavelet decomposition, Represents the low-frequency information of the right view after the first wavelet decomposition; Step 2.2, the left view image After the first downsampling operation and one convolution operation, we get , as shown in formula (3), the obtained features are combined with the low-frequency information obtained by the first wavelet decomposition Splicing along the channel dimension and performing a convolution operation to obtain , as shown in formula (4); the operation of the right view is the same as that of the left view, and the processing process is shown in formulas (5) and (6): (3) (4) (5) (6) in, represents the downsampling operation, Represents a splicing operation, Represents the left view feature obtained after the splicing operation, Represents the right view features obtained after the stitching operation; Step 2.3, the left view feature obtained in step 2.2 Sent to the lighting adjustment block, the left view feature After layer normalization and large kernel convolution operations, it then enters the gating mechanism and convolution layer to obtain , as shown in formula (7); The residual connection is performed with the initial input feature map. The features after the residual connection are normalized once again and then sent to the channel attention module to extract channel features. Then, they are output again through the gating module and convolution layer. , as shown in formula (8), the operation of the right view is the same as that of the left view, and the processing process is shown in formulas (9) and (10): (7) (8) (9) (10) in, SG represents the gating block, represents large kernel convolution, Representation layer normalization, represents channel attention, is the output left view feature, is the output right view feature; Step 2.4: The left view features output in step 2.3 It is then concatenated with the features after the second downsampling and then fed into the wavelet transform to separate the high-frequency and low-frequency information. , as shown in formula (11), the separated low-frequency information is sent to the illumination adjustment module again to obtain the feature , as shown in formula (12), the right view and the left view are operated in the same way, as shown in formulas (13) and (14): (11) (12) (13) (14) in, Represents the low-frequency information of the left view separated for the second time, Represents the low-frequency information of the right view separated for the second time, is the high-frequency information of the left view in different horizontal directions separated for the second time, is the high-frequency information of the left view in the vertical direction separated for the second time, is the high-frequency information of the left view in the diagonal direction separated for the second time; is the high-frequency information of the right view in the horizontal direction separated for the second time, is the high-frequency information of the right view in the vertical direction separated for the second time, is the high-frequency information of the right view in the diagonal direction separated for the second time, Represents the left view features after the lighting adjustment block, Represents the right view features after the lighting adjustment block; Step 2.5: The left view features after the illumination adjustment block in step 2.4 are After being spliced ​​with the features after the third downsampling, it is then sent to the wavelet transform to separate the high-frequency and low-frequency information. , as shown in formula (15), the separated low-frequency information is sent to the illumination adjustment module again to obtain the feature , as shown in formula (16), the operation of the right view is the same as that of the left view, as shown in formulas (17) and (18): (15) (16) (17) (18) in: Represents the low-frequency information of the left view separated for the third time, Represents the low-frequency information of the right view separated for the third time, is the high-frequency information of the left view in the horizontal direction separated for the third time, is the high-frequency information of the left view in the vertical direction separated for the third time, It is the high-frequency information of the left view in the diagonal direction separated for the third time; is the high-frequency information of the right view in the horizontal direction separated for the third time; is the high-frequency information of the right view in the vertical direction separated for the third time; It is the high-frequency information of the right view in the diagonal direction separated for the third time; Represents the left view features after the lighting adjustment block, Represents the features of the right view after the lighting adjustment block; In step 2.6, the high-frequency information of the left and right views obtained in steps 2.3 to 2.5 is input into the high-frequency guided stereo matching module for feature fusion. After fusion, a normalization layer and a linear layer are cascaded to obtain normalized features, as shown in the following formulas (19) and (20): (19) (20) in: For the fusion module, represents a linear layer, represents the normalization layer, Represents the left view features obtained after processing, Represents the right view features obtained after processing; Step 2.7, the matrix obtained in step 2.6 With the matrix After the matrix multiplication operation, the result is then passed through Function processing, finally get the score matrix , where the score matrix S represents the mapping score matrix from the right view to the left view, and the transposed matrix It represents the mapping score matrix from the left view to the right view, as shown in formula (21): (twenty one) Step 2.8: High-frequency information of different directions of the right view The score matrix obtained in step 2.7 Perform matrix multiplication to obtain the mapping of the right view features to the left view features, as shown in formula (22), and convert the high-frequency information of different directions of the left view into With the score matrix Perform matrix multiplication to obtain the mapping from left view features to right view features, as shown in formula (23): (twenty two) (twenty three) Step 2.9, the obtained left view mapping matrix Multi-directional high-frequency information features with the right view input Perform element-by-element addition operation to obtain high-frequency information features after cross-scale interaction processing, as shown in formulas (24) to (26); perform element-by-element addition operation on the right view and the left view. Figure 1The same operation is shown in formulas (27) to (29): (twenty four) (25) (26) (27) (28) (29) in, are the high-frequency information features in the horizontal, vertical and diagonal directions obtained from the right view after cross-view interaction, They are the high-frequency information features in the horizontal, vertical and diagonal directions obtained from the left view after cross-view interaction; Step 2.10: The high-frequency information in three different directions after the left view interaction in step 2.9 is The input details and texture enhancement modules undergo a depthwise separable convolution respectively, and then enter the selection feature fusion module to fuse the three features to obtain , as shown in formula (30), the right view is consistent with the left view, as shown in formula (31): (30) (31) in, For the After the wavelet transform and the high-frequency left view information of cross-view interaction, For the After the wavelet transform and the high-frequency right view information of cross-view interaction, is the fused left view feature, is the fused right view feature, is a depth-wise separable convolution, It is the fusion module; Step 2.11: The left view features fused in step 2.10 Cross attention is performed with high-frequency information in the horizontal and vertical directions respectively, and the fused information is used to supplement the horizontal and vertical information to obtain , as shown in formula (32), the operation of the right view is the same as that of the left view, as shown in formula (33): (32) (33) in, For cross attention, The high-frequency information in the horizontal and vertical directions of the supplemented left view is It is the high-frequency information in the horizontal and vertical directions of the supplemented right view; Step 2.12: The high-frequency information of the horizontal and vertical directions of the left view supplemented in step 2.11 is and the high-frequency information on the original diagonal Perform a cross attention respectively, then splice the supplementary results along the channel dimension, and use convolution to restore the number of channels to obtain features , as shown in formula (34), the right view is operated in the same way as the left view, as shown in formula (35): (34) (35) in, is the supplemented high-frequency information of the diagonal line of the left view, It is the supplemented high-frequency information of the diagonal line of the right view; Step 2.13, the high-frequency information of the left view on the horizontal, vertical and diagonal lines supplemented in steps 2.11 and 2.12 ( ) After a depth-wise separable convolution, the enhanced result of the left view is obtained ( ), the right view is operated in the same way as the left view. After a depth-wise separable convolution, the enhanced result of the right view is obtained. ,in, ; Step 2.14: Convert the low-frequency information of the left view after color adjustment in step 2.5 to and the high-frequency information after detail enhancement in step 2.13 ( ) The first reconstruction is performed using the inverse wavelet transform, as shown in formula (36). The operation of the right view is consistent with that of the left view, as shown in formula (37): (36) (37) in, IDWT is the inverse wavelet transform operation, To enhance the high-frequency information of the left view after three wavelet transforms, To enhance the high-frequency information of the right view after three wavelet transforms, The low-frequency information of the left view after color correction is obtained by three wavelet transforms. The low-frequency information of the right view after color correction is obtained by three wavelet transforms. It is a low-frequency information feature enhancement module. This is the left view result of the first reconstruction. This is the right view result of the first reconstruction; Step 2.15: Based on the result of the first reconstruction in step 2.14, the high-frequency information of the left view is reconstructed for the second time using wavelet transform, as shown in formula (38). The operation of the right view is the same as that of the left view, as shown in formula (39): (38) (39) in, This is the result of the second reconstruction of the left view. This is the result of the second reconstruction of the right view; Step 2.16: Based on the result of the second reconstruction in step 2.15, the high-frequency information of the left view is reconstructed for the third time using wavelet transform. After the reconstruction, the image is sent to the 3×3 convolution kernel, the number of channels is adjusted, and it is spliced ​​with the original view sent to the network to finally output the image. , as shown in formula (40), the right view is operated in the same way as the left view, as shown in formula (41): (40) (41) in, is the original left view fed into the network, is the final enhanced left view, is the original right view fed into the network, This is the final enhanced right view.

[0008] During the training process of step 3, five loss functions are used to train the dark light stereo image enhancement network model based on wavelet decoupling constructed in step 2, namely the loss function in the frequency domain, the loss function in the spatial domain, the frequency domain loss function for the low-frequency information after wavelet transformation, and the spatial domain loss function. Loss function.

[0009] In step 3, the total loss function during training is As shown in the following formula (42): (42) in, is the loss function in the frequency domain, is the loss function in the spatial domain, is the frequency domain loss function of the low-frequency information after wavelet transformation, is the spatial domain loss function of the low-frequency information after wavelet transformation, is the low-frequency information after wavelet transformation Loss function.

[0010] During the training process of step 3, when calculating the loss function in the frequency domain, the enhanced image output by the network and the low-frequency component obtained by the three-level wavelet decomposition are first Fourier transformed, and then converted to the frequency domain space; then the two sets of spectral features are applied in the frequency domain. Norm constraint, construct the frequency domain consistency loss function, as shown in formulas (43) and (44): (43) (44) in, represents the fast Fourier transform, express loss, and Represents the real left and right views, and represents the enhanced left and right views, and The left and right views represent the low-frequency features after the real cubic wavelet transform. and Left and right views representing the low-frequency features decoupled from the network.

[0011] During the training process in step 3, the complement form of the structural similarity index is used in the calculation of the loss function in the spatial domain. That is, the structural similarity index between the enhanced image and the reference image is calculated, and then the value is subtracted from 1 as the final loss metric, as shown in formulas (45) and (46): (45) (46) in, is the structural similarity function.

[0012] During the training process of step 3, the low-frequency information after the three-wavelet transform is used The loss is constrained, as shown in formula (47): (47) in: for loss.

[0013] The present invention achieves the beneficial effect of separating the input image into low-frequency and high-frequency components through three-level wavelet decomposition. The low-frequency information undergoes color correction via a feature enhancement module, while the high-frequency information achieves detail enhancement via an innovative cross-view attention mechanism. Specifically, the method first performs channel expansion and wavelet decomposition on the left and right views, respectively, and constructs a bidirectional mapping matrix to achieve cross-view interaction of high-frequency features. It then employs depthwise separable convolution and cross-attention to perform multi-level fusion and enhancement of high-frequency features in three directions. Finally, the image is gradually reconstructed through a three-level inverse wavelet transform, with a low-frequency enhancement module introduced after each reconstruction level for color correction. Experiments demonstrate that this method can effectively restore normal illumination and detailed texture in images, with excellent performance in metrics such as peak signal-to-noise ratio (PSNR). Key innovations include: 1) a multi-scale decomposition and progressive reconstruction framework; 2) cross-view high-frequency feature interaction based on an attention mechanism; and 3) a collaborative optimization strategy for low-frequency color correction and high-frequency detail enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the exchange of low-frequency wavelet coefficients between low-light images and normal-light images; Figure 2 It is the peak signal-to-noise ratio distribution diagram of the original low-light image and the restored low-light image, and the original normal-light image and the restored normal-light image; Figure 3 Schematic diagram of the dark-light stereo image enhancement network structure based on wavelet decoupling constructed in the low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling of the present invention; Figure 4 This is an architectural diagram of the illumination adjustment module (IAM) in the low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling of the present invention; Figure 5 This is an architecture diagram of the high-frequency guided stereo matching module (HF-CIM) in the low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling of the present invention; Figure 6 This is an architecture diagram of the detail and texture enhancement module (DTEM) in the low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling of the present invention; Figure 7 This is a qualitative result diagram of the image enhanced by the low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling of the present invention on the Kitti2015 dataset; Figure 8 This is a qualitative result diagram of an image enhanced by the low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling of the present invention on the Flickr dataset. DETAILED DESCRIPTION

[0015] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] The present invention is a low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling. It utilizes the idea of ​​multi-scale wavelet domain decoupling to independently process high-frequency and low-frequency information, perform cross-view information interaction on high-frequency detail information, and restore color for low-frequency information.

[0017] Before the formal experiment, we conducted multiple wavelet decomposition experiments to verify that after the image was transformed multiple times, the illumination information was still retained in the low-frequency information, and the detail texture information was retained in the high-frequency information. This prepared for the subsequent design of two independent branches to process the illumination and detail information respectively. The specific details are as follows: like Figure 1 As shown, we first select a pair of images from the classic monocular low-light dataset LOLv1, which are low-light images and normal light images (3 represents the number of channels of the image, H is the height of the image, and W is the width of the image). The two images are decomposed three times using wavelet transform. Taking the normal light image as an example, the first wavelet transform is Then the normal light low frequency information will be separated and normal light high-frequency information in 3 different directions (horizontal, vertical, diagonal) , perform a second wavelet transform on the separated low-frequency information , separate the low-frequency information and high-frequency information , and then perform the third wavelet transform on the low-frequency information separated for the second time , separate the low-frequency information and high-frequency information , the low-light image will be decomposed into low-frequency and high frequency ( is low-frequency information, for high-frequency information).

[0018] The low-frequency information of the normal light image separated by three wavelet transforms , and the high-frequency information of the low-light image separated by three wavelet transforms Combine and perform 3 inverse wavelet transforms Reconstructed image ; Then the low-frequency information of the low-light image separated by three wavelet transforms , and the high-frequency information of the normal light image separated by three wavelet transforms Combine and perform 3 inverse wavelet transforms Reconstructed image . The reconstructed image was found Restore to normal color and reconstruct the image Still low light.

[0019] like Figure 2 As shown, 100 pairs of normal light and low light images were selected from the LOLv1 dataset to repeat the above experiment, and then the PSNR (peak signal-to-noise ratio) was used to reconstruct the normal light images. and the original normal light image , the reconstructed low-light image and original low-light image The test was carried out and the measured values ​​were all normal.

[0020] Complete preliminary experimental verification.

[0021] Example 1 The present invention provides a low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling, which specifically includes the following steps: Step 1, Dataset Preparation: Select three binocular stereo image datasets, Flickr2014, Holopix50, and KITTI 2015, as experimental data. Divide them into training, validation, and test sets in a reasonable proportion. Generate corresponding low-light versions based on the original images for image enhancement experiments under low-light conditions. Step 2: Build a model: Construct a dark-light stereo image enhancement network based on wavelet decoupling (WDCI-Net), which includes five modules: wavelet transform and downsampling module, illumination adjustment module, high-frequency guided stereo matching module, high-frequency information feature enhancement module, and inverse wavelet transform image reconstruction module. Step 3, training phase: use the training set preprocessed in step 1 to train the network model constructed in step 2 to obtain the optimal enhanced network model; Step 4, testing phase: After the network is trained, the training weights are obtained. The test data in the test set are tested using the trained weights to obtain quantitative and qualitative test results.

[0022] Example 2 The specific process of step 1 is: In step 1.1, during the dataset preparation process, we selected 1,225 pairs of stereo images for synthetic experimental data. These included 762 pairs from the Holopix50 dataset, 238 pairs from the Flickr2014 dataset, and 225 pairs from the KITTI2015 dataset. These images covered a variety of typical scenes. We rationally divided the selected images into 901 pairs for training, 24 pairs for validation, and 300 pairs for testing, ensuring the scientific and comprehensive nature of model training and performance evaluation.

[0023] Step 1.2: Since the existing commonly used stereo matching datasets do not contain images collected under low-light conditions, in order to meet the experimental requirements of the low-light stereo image enhancement task, this paper selects stereo image pairs under normal lighting conditions and generates their corresponding synthetic low-light versions. The image selection process is based on subjective image quality assessment and combines two non-reference image quality evaluation indicators: natural image quality evaluation index and natural image quality no-reference quality assessment metrics When the image meets and When the condition is met, the image quality is determined to be high and can be used for subsequent low-light synthesis processing. Specifically, to generate a synthetic low-light image, the gamma correction method is used to adjust the brightness of the original image to reduce the overall brightness level, thereby simulating the visual effect in a low-light environment. As shown in formula (1): (1) in: is the input image, is the low-light image obtained ( Can represent and , and left view and right view respectively), 、 and They are the parameters for adjusting the lighting.

[0024] Example 3 The specific process of step 2 is: Step 2.1, as Figure 3 As shown, first input low-light binocular stereo images, which are the left view and right view (3 represents the number of channels of the image, is the height of the image, is the width of the image), taking the left view as an example, we first use 3×3 convolution to expand the image from 3 channels to 48 channels. , and then perform wavelet transform on the expanded features to obtain the features , as shown in formula (2), the right view is consistent with the left view, as shown in formula (3), (2) (3) Where: DWT represents the wavelet transform operation, Represents the high-frequency information in the horizontal, vertical and diagonal directions of the left view after the first wavelet decomposition. Represents the low-frequency information of the left view after the first wavelet decomposition, It represents the high-frequency information in the horizontal, vertical and diagonal directions of the right view after the first wavelet decomposition. Represents the low-frequency information of the right view after the first wavelet decomposition; Step 2.2, taking the left view as an example, input low-light binocular stereo image After the first downsampling operation and one convolution operation, we get , as shown in formula (4), the obtained features are combined with the low-frequency information obtained by the first wavelet decomposition Splicing is performed along the channel dimension, and a convolution operation is performed to restore it to 48 channels, resulting in , as shown in formula (5). The operation of the right view is the same as that of the left view, as shown in formula (6) (7): (4) (5) (6) (7) in, represents the downsampling operation, Represents a splicing operation, Represents the left view feature obtained after the splicing operation, Represents the right view features obtained after the stitching operation; Step 2.3, taking the left view as an example, the left view features obtained in step 2.2 are The next step is to feed the illumination adjustment block (IAM), such as Figure 4 As shown in the figure, the illumination adjustment is fast and consists of layer normalization, large kernel convolution, gating mechanism, channel attention, and convolution layer. The feature map is sequentially subjected to layer normalization and large kernel convolution operations, and then enters the gating mechanism and convolution layer to obtain , as shown in formula (8). The output of this module is residually connected with the initial input feature map. The features after residual connection are normalized again, and then sent to the channel attention module to extract channel features, and then passed through the gating module and convolution layer again, and the final output result is , as shown in formula (9), the right view is consistent with the left view, as shown in formula (10) (11): (8) (9) (10) (11) in, SG represents the gating block, represents large kernel convolution, Representation layer normalization, represents channel attention, Output the result for the left view, Output the result for the left view; Step 2.4, taking the left view as an example, the features output in step 2.3 It is then concatenated with the features after the second downsampling and then fed into the wavelet transform to separate the high-frequency and low-frequency information. , as shown in formula (12), the separated low-frequency information is sent to the illumination adjustment module again to obtain the feature , as shown in formula (13), the right view and the left view operations are the same, as shown in formulas (14) and (15): (12) (13) (14) (15) in: Represents the low-frequency information of the left view separated for the second time, Represents the low-frequency information of the right view separated for the second time, is the high-frequency information of the left view in three different directions separated for the second time, and is the high-frequency information of the right view in three different directions separated for the second time, Represents the left view features after the lighting adjustment block, Represents the right view features after the lighting adjustment block; Step 2.5, taking the left view as an example, the features after the illumination adjustment block in step 2.4 are After being spliced ​​with the features after the third downsampling, it is then sent to the wavelet transform to separate the high-frequency and low-frequency information. , as shown in formula (16), the separated low-frequency information is sent to the illumination adjustment module again to obtain the feature , as shown in formula (17), the operation of the right view is the same as that of the left view, as shown in formulas (18) and (19): (16) (17) (18) (19) in: Represents the low-frequency information of the left view separated for the third time, Represents the low-frequency information of the right view separated for the third time, is the high-frequency information of the left view in three different directions separated for the third time, is the high-frequency information of the right view in three different directions separated for the third time, Represents the characteristics of the left view after the lighting adjustment block, Represents the features of the right view after the lighting adjustment block; Step 2.6, as Figure 5 As shown in the figure, the proposed high-frequency guided stereo matching module (HF-CIM) contains a selection feature fusion module (SKFF), a normalization layer, and a linear layer, which separates the high-frequency information of the left and right views through each layer of wavelet transform. and ,First, feature fusion is performed, and after fusion, a normalization layer and a linear layer are cascaded to obtain normalized features. The left view and right view are specifically shown in formulas (20) and (21): (20) (twenty one) in, For the fusion module, represents a linear layer, represents the normalization layer, Respectively represent the left and right view features obtained after processing; Step 2.7, the matrix obtained in step 2.6 With the matrix After the matrix multiplication operation, the result is then passed through Function processing, finally get the score matrix Among them, the score matrix S represents the mapping score matrix from the right view to the left view, and the transposed matrix It represents the mapping score matrix from the left view to the right view, as shown in formula (22): (twenty two) Step 2.8: High-frequency information of different directions of the right view and the score matrix obtained in 2.7 Perform matrix multiplication to obtain the mapping of the right view feature to the left view feature, as shown in formula (23), and convert the high-frequency information of different directions of the left view into With the score matrix Perform matrix multiplication to obtain the mapping from left view features to right view features, as shown in formula (24): (twenty three) (twenty four) Step 2.9, taking the left view as an example, the obtained mapping matrix Multi-directional high-frequency information features with the right view input Perform element-by-element addition to obtain high-frequency information features after cross-scale interaction processing, as shown in formulas (25) to (27). The operations for the right view and the left view are the same, as shown in formulas (28) to (30): (25) (26) (27) (28) (29) (30) in, The high-frequency information features obtained from the right view after cross-view interaction, It is the high-frequency information feature obtained from the left view after cross-view interaction.

[0025] Step 2.10, as Figure 6 As shown, the detail and texture enhancement module (DTEM) contains a depth-separable convolution and a selection feature fusion module. And cross attention, taking the left view as an example, after cross-view interaction, the high-frequency information in three different directions after the interaction in step 2.9 is After a depth-wise separable convolution, the features are further extracted and then sent to the selection feature fusion module to fuse the three features to obtain , as shown in formula (31), the right view is operated in the same way as the left view, as shown in formula (32): (31) (32) in, For the After the wavelet transform and the high-frequency left view information of cross-view interaction, For the After the wavelet transform and the high-frequency right view information of cross-view interaction, is the fused left view feature, is the fused right view feature, is a depth-wise separable convolution, It is the fusion module; Step 2.11, taking the left view as an example, the features fused in step 2.10 First, cross-attention is performed with the high-frequency information in the horizontal and vertical directions, and the fused information is used to supplement the horizontal and vertical information to obtain , as shown in formula (33), the operation of the right view is the same as that of the left view, as shown in formula (34): (33) (34) in, For cross attention, The high-frequency information in the horizontal and vertical directions of the supplemented left view is It is the high-frequency information in the horizontal and vertical directions of the supplemented right view; Step 2.12, taking the left view as an example, the high-frequency information in the horizontal and vertical directions supplemented in step 2.11 is and the high-frequency information on the original diagonal Perform a cross attention respectively, then splice the supplementary results along the channel dimension, and use 3×3 convolution to restore the number of channels to obtain features , as shown in formula (35), the right view is operated in the same way as the left view, as shown in formula (36): (35) (36) in, is the supplemented high-frequency information of the diagonal line of the left view, is the supplemented high-frequency information of the diagonal line of the right view, for convolution.

[0026] Step 2.13, taking the left view as an example, the high-frequency information of the left view on the horizontal, vertical and diagonal lines supplemented in steps 2.11 and 2.12 ( ) After a depth-wise separable convolution, the final enhanced result is obtained The right view is the same as the left view, and after a depth-wise separable convolution, it is obtained .

[0027] Step 2.14, taking the left view as an example, convert the low-frequency information after color adjustment in steps 2.3, 2.4 and 2.5 and the high-frequency information after detail enhancement in step 2.13 ( ) The first reconstruction is performed using the inverse wavelet transform, as shown in formula (37). The operation of the right view is consistent with that of the left view, as shown in formula (38): (37) (38) in, IDWT is the inverse wavelet transform operation, The high-frequency information of the left view is enhanced after three wavelet transforms. The high-frequency information of the right view is enhanced after three wavelet transforms. The low-frequency information of the left view after color correction by three wavelet transforms is The low-frequency information of the right view after color correction by three wavelet transforms is It is a low-frequency information feature enhancement module. This is the left view result of the first reconstruction. This is the right view result of the first reconstruction.

[0028] In step 2.15, taking the left view as an example, a second reconstruction is performed, as shown in formula (39). The operation of the right view is the same as that of the left view, as shown in formula (40): (39) (40) in, This is the result of the second reconstruction of the left view. This is the result of the second reconstruction of the right view; Step 2.16, taking the left view as an example, perform the third reconstruction. After the reconstruction, send the image into the 3×3 convolution kernel, adjust the number of channels to 3, and splice it with the original view sent to the network to finally output the image. , as shown in formula (41), the right view operation is consistent with the left view, as shown in formula (42): (41) (42) in, is the original left view fed into the network, is the final enhanced left view, is the original right view fed into the network, This is the final enhanced right view.

[0029] Example 4 Step 3 is as follows: In step 3.1, the processed data from step 1 was fed into the dark-light enhancement network built in step 2. All experiments were conducted using the PyTorch framework on a device equipped with an RTX 3090 GPU. The Adam optimizer was used, with a batch size of 20. All training images were randomly cropped to 128×128 pixels. The initial learning rate was set to 0.0002 and halved every 250 training epochs. The total training time for the entire network was 1000 epochs.

[0030] In step 3.2, after each training cycle, the model weights obtained from the current training cycle are used to perform augmented validation on 20 pairs of pre-set stereo images. The validation set images are uniformly cropped to 400×400 pixels (batch size 1). This process does not generate visual results; only quantitative evaluation metrics are output, including peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). By tracking and analyzing these quantitative metrics, the degree of fit and performance evolution during network training can be effectively assessed.

[0031] Step 3.3, five loss functions are used to guide model training, namely the loss in the frequency domain and the loss in the spatial domain, as well as the frequency domain loss, spatial domain loss and Loss, as shown in formula (43): (43) in, is the loss in the frequency domain, is the loss in the spatial domain, e is the frequency domain loss of low-frequency information after three wavelet transforms, is the spatial domain loss of low-frequency information after three wavelet transforms, is the low-frequency information after 3 wavelet transforms loss.

[0032] In terms of frequency domain loss calculation, the enhanced image output by the network and the low-frequency component obtained by three-level wavelet decomposition are first Fourier transformed to convert them into frequency domain space; then, the two sets of spectral features are applied in the frequency domain. Norm constraint, construct the frequency domain consistency loss function, as shown in formulas (44) and (45): (44) (45) in, represents the fast Fourier transform, express loss, and Represents the real left and right views, and represents the enhanced left and right views, and The left and right views represent the low-frequency features after the real 3-times wavelet transform. and Left and right views representing the low-frequency features decoupled from the network.

[0033] In the calculation of spatial domain loss, the complement form of the structural similarity index is used, that is, the structural similarity index between the enhanced image and the reference image is calculated. , and then subtract this value from 1 as the final loss metric, as shown in Formula (46) and Formula (47): (46) (47) in, is the structural similarity function.

[0034] For the low-frequency information after the 3rd wavelet transform, use The loss is constrained, as shown in formula (48): (48) in: for loss.

[0035] Example 5 The specific process of step 4 is: After fixing the network weight parameters trained in step 3, we tested and verified them on the test sets of two public benchmark datasets, Flickr and Kitti. The Flickr test set mainly contains rich natural scenes, while the Kitti test set focuses on visual data of autonomous driving scenarios. During the test, we obtained quantitative and qualitative results of the two datasets. From the analysis of the results, we found that the trained weights showed good illumination adjustment capabilities on both test sets, but there is still room for improvement in detail processing in complex scenes. The visualization results of the Kitti dataset are shown below. Figure 7 As shown in the figure, it can be seen that the original left view and the original right view are affected by low illumination, the overall picture is dim, and details such as buildings and vehicles are submerged in shadows, with low recognition; the enhanced left view and right view have improved brightness, the picture is clear and bright, and details such as building textures and street elements are presented, and the visual effect is significantly improved.

[0036] Example 6 The visualization results of the Flickr dataset are as follows Figure 8 As shown in the figure, the original left and right views appear dim and unclear due to low illumination. The enhanced left and right views have increased brightness, resulting in clearer images, more detailed images of buildings and vehicles, and significantly improved visual quality. The quantitative evaluation metrics for the test are shown in Table 1; Table 1 shows the quantitative results on the Flickr and Kitti2015 datasets.

[0037] Table 1

[0038] This method, based on multi-scale wavelet domain decoupling, implements a low-light binocular stereo image enhancement method. First, the input low-light image undergoes multi-scale wavelet decomposition to accurately separate the image's high-frequency information (texture) and low-frequency information (illumination). Two independent branches are then designed to perform targeted processing on low-frequency illumination information and high-frequency texture information, respectively. The parallax attention mechanism is cleverly applied to high-frequency texture information at different scales, effectively improving the ability to capture texture details. This fused information is then input into a high-frequency information enhancement module for optimized detail processing. Ultimately, this series of sophisticated operations successfully restores a high-quality image. This method effectively addresses the complex degradation and severe noise issues inherent in existing low-light enhancement methods, while also overcoming the black-box problem caused by the highly entangled enhancement features within a single latent space.

Claims

1. A low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling, characterized by: The specific steps include: Step 1: Select a binocular stereo image pair dataset, perform low-light preprocessing on the image pairs in the dataset to obtain a low-light binocular stereo image pair dataset, and divide the preprocessed dataset into a training set, a validation set, and a test set; Step 2: Construct a dark-light stereo image enhancement network based on wavelet decoupling; Step 3: Use the training set preprocessed in step 1 to train the network model constructed in step 2 to obtain the optimal enhanced network model; Step 4: Test the enhanced network model obtained in step 3 using the test set preprocessed in step 1, and output the enhanced image.

2. The low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling according to claim 1, characterized in that: The dark light stereo image enhancement network based on wavelet decoupling constructed in step 2 includes: a wavelet transform and downsampling module, an illumination adjustment module, a high-frequency guided stereo matching module, a high-frequency information feature enhancement module and an inverse wavelet transform image reconstruction module.

3. The low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling according to claim 2, characterized in that: The specific process of step 2 is: Step 2.1, first input low-light binocular stereo images, which are the left view and right view , first use 3×3 convolution to expand the channel of the left view image, and get , and then perform wavelet transform on the expanded left view features to obtain the low-frequency information of the left view after the first wavelet decomposition , as shown in formula (1), the right view is operated in the same way as the left view, as shown in formula (2): （1） （2） Among them, DWT represents the wavelet transform operation, Represents the high-frequency information in the horizontal direction of the left view after the first wavelet decomposition, Represents the high-frequency information in the vertical direction of the left view after the first wavelet decomposition, Represents the high-frequency information in the diagonal direction of the left view after the first wavelet decomposition, Represents the low-frequency information of the left view after the first wavelet decomposition; Represents the high-frequency information in the horizontal direction of the right view after the first wavelet decomposition, Represents the high-frequency information in the vertical direction of the right view after the first wavelet decomposition, Represents the high-frequency information in the diagonal direction of the right view after the first wavelet decomposition, Represents the low-frequency information of the right view after the first wavelet decomposition; Step 2.2, the left view image After the first downsampling operation and one convolution operation, we get , as shown in formula (3), the obtained features are combined with the low-frequency information obtained by the first wavelet decomposition Splicing along the channel dimension and performing a convolution operation to obtain , as shown in formula (4); the operation of the right view is the same as that of the left view, and the processing process is shown in formulas (5) and (6): （3） （4） （5） （6） in, represents the downsampling operation, Represents a splicing operation, Represents the left view feature obtained after the splicing operation, Represents the right view features obtained after the stitching operation; Step 2.3, the left view feature obtained in step 2.2 Sent to the lighting adjustment block, the left view feature After layer normalization and large kernel convolution operations, it then enters the gating mechanism and convolution layer to obtain , as shown in formula (7); The residual connection is performed with the initial input feature map. The features after the residual connection are normalized once again and then sent to the channel attention module to extract channel features. Then, they are output again through the gating module and convolution layer. , as shown in formula (8), the operation of the right view is the same as that of the left view, and the processing process is shown in formulas (9) and (10): （7） （8） （9） （10） in, SG represents the gating block, represents large kernel convolution, Representation layer normalization, represents channel attention, is the output left view feature, is the output right view feature; Step 2.4: The left view features output in step 2.3 It is then concatenated with the features after the second downsampling and then fed into the wavelet transform to separate the high-frequency and low-frequency information. , as shown in formula (11), the separated low-frequency information is sent to the illumination adjustment module again to obtain the feature , as shown in formula (12), the right view and the left view are operated in the same way, as shown in formulas (13) and (14): （11） （12） （13） （14） in, Represents the low-frequency information of the left view separated for the second time, Represents the low-frequency information of the right view separated for the second time, is the high-frequency information of the left view in different horizontal directions separated for the second time, is the high-frequency information of the left view in the vertical direction separated for the second time, is the high-frequency information of the left view in the diagonal direction separated for the second time; is the high-frequency information of the right view in the horizontal direction separated for the second time, is the high-frequency information of the right view in the vertical direction separated for the second time, is the high-frequency information of the right view in the diagonal direction separated for the second time, Represents the left view features after the lighting adjustment block, Represents the right view features after the lighting adjustment block; Step 2.5: The left view features after the illumination adjustment block in step 2.4 are After being spliced ​​with the features after the third downsampling, it is then sent to the wavelet transform to separate the high-frequency and low-frequency information. , as shown in formula (15), the separated low-frequency information is sent to the illumination adjustment module again to obtain the feature , as shown in formula (16), the operation of the right view is the same as that of the left view, as shown in formulas (17) and (18): （15） （16） （17） （18） in: Represents the low-frequency information of the left view separated for the third time, Represents the low-frequency information of the right view separated for the third time, is the high-frequency information of the left view in the horizontal direction separated for the third time, is the high-frequency information of the left view in the vertical direction separated for the third time, It is the high-frequency information of the left view in the diagonal direction separated for the third time; is the high-frequency information of the right view in the horizontal direction separated for the third time; is the high-frequency information of the right view in the vertical direction separated for the third time; It is the high-frequency information of the right view in the diagonal direction separated for the third time; Represents the left view features after the lighting adjustment block, Represents the features of the right view after the lighting adjustment block; In step 2.6, the high-frequency information of the left and right views obtained in steps 2.3 to 2.5 is input into the high-frequency guided stereo matching module for feature fusion. After fusion, a normalization layer and a linear layer are cascaded to obtain normalized features, as shown in the following formulas (19) and (20): （19） （20） in: For the fusion module, represents a linear layer, represents the normalization layer, Represents the left view features obtained after processing, Represents the right view features obtained after processing; Step 2.7, the matrix obtained in step 2.6 With the matrix After the matrix multiplication operation, the result is then passed through Function processing, finally get the score matrix , where the score matrix S represents the mapping score matrix from the right view to the left view, and the transposed matrix It represents the mapping score matrix from the left view to the right view, as shown in formula (21): （21） Step 2.8: High-frequency information of different directions of the right view The score matrix obtained in step 2.7 Perform matrix multiplication to obtain the mapping of the right view features to the left view features, as shown in formula (22), and convert the high-frequency information of different directions of the left view into With the score matrix Perform matrix multiplication to obtain the mapping from left view features to right view features, as shown in formula (23): （22） （23） Step 2.9, the obtained left view mapping matrix Multi-directional high-frequency information features with the right view input Perform element-by-element addition to obtain high-frequency information features after cross-scale interaction processing, as shown in formulas (24) to (26). Perform the same operation on the right view as on the left view, as shown in formulas (27) to (29): （24） （25） （26） （27） （28） （29） in, are the high-frequency information features in the horizontal, vertical and diagonal directions obtained from the right view after cross-view interaction, They are the high-frequency information features in the horizontal, vertical and diagonal directions obtained from the left view after cross-view interaction; Step 2.10: The high-frequency information in three different directions after the left view interaction in step 2.9 is The input details and texture enhancement modules undergo a depthwise separable convolution respectively, and then enter the selection feature fusion module to fuse the three features to obtain , as shown in formula (30), the right view is consistent with the left view, as shown in formula (31): （30） （31） in, For the After the wavelet transform and the high-frequency left view information of cross-view interaction, For the After the wavelet transform and the high-frequency right view information of cross-view interaction, is the fused left view feature, is the fused right view feature, is a depth-wise separable convolution, It is the fusion module; Step 2.11: The left view features fused in step 2.10 Cross attention is performed with high-frequency information in the horizontal and vertical directions respectively, and the fused information is used to supplement the horizontal and vertical information to obtain , as shown in formula (32), the operation of the right view is the same as that of the left view, as shown in formula (33): （32） （33） in, For cross attention, The high-frequency information in the horizontal and vertical directions of the supplemented left view is It is the high-frequency information in the horizontal and vertical directions of the supplemented right view; Step 2.12: The high-frequency information of the horizontal and vertical directions of the left view supplemented in step 2.11 is and the high-frequency information on the original diagonal Perform a cross attention respectively, then splice the supplementary results along the channel dimension, and use convolution to restore the number of channels to obtain features , as shown in formula (34), the right view is operated in the same way as the left view, as shown in formula (35): （34） （35） in, is the supplemented high-frequency information of the diagonal line of the left view, It is the supplemented high-frequency information of the diagonal line of the right view; Step 2.13, the high-frequency information of the left view on the horizontal, vertical and diagonal lines supplemented in steps 2.11 and 2.12 ( ) After a depth-wise separable convolution, the enhanced result of the left view is obtained ( ), the right view is operated in the same way as the left view. After a depth-wise separable convolution, the enhanced result of the right view is obtained. ,in, ; Step 2.14: Convert the low-frequency information of the left view after color adjustment in step 2.5 to and the high-frequency information after detail enhancement in step 2.13 ( ) The first reconstruction is performed using the inverse wavelet transform, as shown in formula (36). The operation of the right view is consistent with that of the left view, as shown in formula (37): （36） （37） in, IDWT is the inverse wavelet transform operation, To enhance the high-frequency information of the left view after three wavelet transforms, To enhance the high-frequency information of the right view after three wavelet transforms, The low-frequency information of the left view after color correction is obtained by three wavelet transforms. The low-frequency information of the right view after color correction is obtained by three wavelet transforms. It is a low-frequency information feature enhancement module. This is the left view result of the first reconstruction. This is the right view result of the first reconstruction; Step 2.15: Based on the result of the first reconstruction in step 2.14, the high-frequency information of the left view is reconstructed for the second time using wavelet transform, as shown in formula (38). The operation of the right view is the same as that of the left view, as shown in formula (39): （38） （39） in, This is the result of the second reconstruction of the left view. This is the result of the second reconstruction of the right view; Step 2.16: Based on the result of the second reconstruction in step 2.15, the high-frequency information of the left view is reconstructed for the third time using wavelet transform. After the reconstruction, the image is sent to the 3×3 convolution kernel, the number of channels is adjusted, and it is spliced ​​with the original view sent to the network to finally output the image. , as shown in formula (40), the right view is operated in the same way as the left view, as shown in formula (41): （40） （41） in, is the original left view fed into the network, is the final enhanced left view, is the original right view fed into the network, This is the final enhanced right view.

4. The low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling according to claim 3, characterized in that: During the training process of step 3, five loss functions are used to train the dark light stereo image enhancement network model based on wavelet decoupling constructed in step 2, namely the loss function in the frequency domain, the loss function in the spatial domain, the frequency domain loss function and the spatial domain loss function for the low-frequency information after wavelet transformation, and Loss function.

5. The low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling according to claim 4, characterized in that: In step 3, the total loss function during training is As shown in the following formula (42): （42） in, is the loss function in the frequency domain, is the loss function in the spatial domain, is the frequency domain loss function of the low-frequency information after wavelet transformation, is the spatial domain loss function of the low-frequency information after wavelet transformation, is the low-frequency information after wavelet transformation Loss function.

6. The low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling according to claim 5, characterized in that: During the training process of step 3, when calculating the loss function in the frequency domain, the enhanced image output by the network and the low-frequency component obtained by the three-level wavelet decomposition are first subjected to Fourier transform respectively, and then converted to the frequency domain space; Then, these two sets of spectral features are applied in the frequency domain Norm constraint, construct the frequency domain consistency loss function, as shown in formulas (43) and (44): （43） （44） in, represents the fast Fourier transform, express loss, and Represents the real left and right views, and represents the enhanced left and right views, and The left and right views represent the low-frequency features after the real cubic wavelet transform. and Left and right views representing the low-frequency features decoupled from the network.

7. The low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling according to claim 6, characterized in that: During the training process of step 3, the complement form of the structural similarity index is used in the calculation of the loss function in the spatial domain. That is, the structural similarity index between the enhanced image and the reference image is calculated, and then the value is subtracted from 1 as the final loss metric, as shown in formulas (45) and (46): （45） （46） in, is the structural similarity function.

8. The low-illumination binocular stereo image enhancement method based on multi-scale wavelet domain decoupling according to claim 7, characterized in that: During the training process of step 3, the low-frequency information after the three-wavelet transformation is used The loss is constrained, as shown in formula (47): （47） in: for loss.

Images