An implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization

Abstract

The present invention discloses an implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization. The method first obtains an RGB image in a normal lighting scene and a depth map of the corresponding viewing angle; and calculates the corresponding three-dimensional point cloud data based on the depth map and the camera intrinsic parameters. The low-light image data is synthesized using the RGB image in the normal lighting scene. The high-frequency features and low-frequency features of the low-light image are decomposed using a high- and low-frequency feature enhancement network to enhance the low-light image, and then the image features and context features of the low-light image are extracted; the 2D-3D feature fusion network is used to extract 2D image features and 3D point cloud features and align and fuse them to obtain the image features and context features of the normal RGB image, which are used to supervise the feature extraction process of the high- and low-frequency feature enhancement network. Finally, a 4D correlation volume table is constructed based on the image features and context features of the low-light image, and the GRU is used to infer the optical flow.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to an implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization. Background Art

[0002] Optical flow estimation aims to compute pixel-level two-dimensional motion between consecutive frames, describing the motion field in two-dimensional space. Traditional optical flow algorithms typically rely on the assumption of photometric consistency and infer motion information through optimization. The core of these methods lies in manual feature extraction, but the introduction of deep learning has significantly changed the approach to optical flow estimation. Early deep learning optical flow methods, such as FlowNet, directly estimate optical flow from image pairs using two convolutional neural networks, eliminating the manual feature extraction step in traditional methods and demonstrating the feasibility of learning optical flow directly from images.

[0003] With the development of deep learning, the design of optical flow estimation networks has become increasingly diverse. For example, PWC-Net adopts a spatial pyramid structure to gradually estimate optical flow at multiple levels. It also introduces a feature pyramid and a cost volume-based matching method to improve the accuracy of optical flow estimation while maintaining computational efficiency. RAFT uses a novel network structure to leverage multi-scale 4D correlation volume lookup and GRU iterative optical flow updates. GMA enhances global motion matching capabilities in feature extraction and encoding through global motion information aggregation and a local attention mechanism, thereby improving the performance of optical flow calculations.

[0004] However, existing optical flow methods are primarily designed for image data collected under normal environmental conditions. In complex environments, image data often suffers from information loss. For example, in low-light conditions, increased noise and weakened texture features significantly degrade image quality, undermining the photometric consistency assumption that optical flow estimation relies on. This increased noise and loss of texture detail reduces texture consistency between frames, leading to reduced performance of the subsequent optical flow estimation network.

[0005] For optical flow estimation in challenging environments such as severe weather conditions or low-light environments, corresponding solutions have also emerged in the existing technology.

[0006] For example, in rainy scenes, RobustNet estimates optical flow by leveraging rain-free residual channels, while RainFlow reduces the impact of rain streaks and fog on image features by generating features that are invariant to rain streaks and fog. In hazy scenes, some methods synthesize haze data and apply style transfer techniques to simultaneously perform image dehazing and optical flow estimation.

[0007] In low-light environments, Zheng et al. proposed a synthetic dark noise optical flow benchmark dataset called the FlyingChairs Dark & ​​Noise (FCDN) dataset, which addresses the dataset missing issue by adding dark image noise to a normal lighting dataset. They also introduced the Various Brightness Optical Flow (VBOF) dataset, which contains images at different exposure levels and optical flow pseudo-labels. Furthermore, CEDFlow introduces high- and low-frequency adaptive enhancement, as well as edge enhancement of feature dimensions, and specifically designs a structure for image feature extraction in low-light environments to improve optical flow estimation performance.

[0008] However, relying solely on the limited information in low-light images to extract image features limits the performance of optical flow estimation. The reduced light intake in low-light conditions makes it difficult to extract necessary information, which hinders the extraction of high-quality features that are critical for the subsequent optical flow calculation. Furthermore, because the network is trained only on low-quality images, this reduces the performance typically achieved under normal lighting conditions. Summary of the Invention

[0009] In response to the shortcomings of the existing technology, the present invention proposes an implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization. By combining RGB and depth image data and introducing multimodal information and a multimodal collaborative optimization framework, the robustness and accuracy of optical flow estimation are significantly improved, solving the shortcomings of the existing technology in difficult scenes such as low light, high noise or other complex dynamic scenes. The method performs particularly well in scenes with low-quality images.

[0010] An implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization includes the following steps:

[0011] Step 1: Obtain an RGB image under normal lighting conditions and a depth map of the corresponding viewing angle. Calculate the 3D point cloud corresponding to each pixel based on the depth map and camera intrinsic parameters.

[0012] Add simulated low-light noise to the RGB image and adjust the image brightness to synthesize low-light image data.

[0013] Step 2: Build a high- and low-frequency feature enhancement network. For the low-light image data obtained in step 1, decompose it into high-frequency features and low-frequency features. The high-frequency features contain image details such as edges and textures, while the low-frequency features are the image background and overall outline.

[0014] The high-frequency features are input into the dense convolutional network for enhancement to highlight the texture details; the low-frequency features are processed by the multi-scale feature enhancement network, and the attention mechanism is used to capture the global background information. Then, the enhanced image features F are generated through the channel attention mechanism and residual connection weighted fusion. en .

[0015] Extracting image features of low-light image data from enhanced image features through an encoder and contextual features .

[0016] Step 3: Use the encoder to extract 2D image features from the RGB image and 3D point cloud under normal lighting scenes respectively. and 3D point cloud features , after feature alignment and fusion, the fused normal illumination image features are obtained and contextual features .

[0017] Through the prior feature loss function Supervise the feature extraction process in step 2:

[0018]

[0019] || ||2 means calculating the 2-norm.

[0020] Step 4: Image features based on low-light images and contextual features Calculate 4D correlation volume , using multi-scale correlation volume, using the recursive update operator of GRU (Gated Recurrent Unit) to iteratively optimize the initial optical flow field, gradually refine the optical flow estimation results, and calculate the optical flow estimation loss .

[0021] Step 5. Set the total loss function is the prior feature loss and optical flow estimation loss The weighted sum of is used to complete the model training:

[0022]

[0023] in, represents the loss weight.

[0024] The image pairs that need to be estimated for optical flow are input into the trained high- and low-frequency feature enhancement network to extract image features. and contextual features , and then output the optical flow estimation result through the method of step 4.

[0025] The present invention has the following beneficial effects:

[0026] 1. Leveraging multimodal data to supplement missing information in original low-quality images, we overcome the performance bottleneck of single-modal optical flow estimation methods in complex scenarios. By decomposing and enhancing high- and low-frequency features, we optimize image detail and global information extraction, respectively, significantly improving the accuracy of optical flow calculations. We introduce an implicit feature supervision mechanism, guiding the training of the enhancement network through multimodal features. By utilizing RGBD fusion and 3D-to-2D projection, we ensure that the enhanced features maintain geometric interpretability, achieving deep collaborative optimization of the task and enhancement.

[0027] 2. This method is applicable to a wide range of computer vision tasks, including robotic visual navigation, autonomous driving environmental perception, and moving object detection and tracking. It performs particularly well in challenging scenarios such as low-light and high-noise environments. Furthermore, this technology has potential value in image restoration, video analysis, and other applications requiring high-precision dynamic scene estimation. It provides a novel solution for optical flow estimation in complex environments. By leveraging implicit multimodal knowledge guidance and feature enhancement, it improves the performance of optical flow estimation in challenging scenarios, further advancing technological advancements in the field of optical flow computation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Flowchart of the implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization.

[0029] Figure 2 This is a qualitative comparison of different methods in the embodiments on the Flyingthing3D dataset. DETAILED DESCRIPTION

[0030] The present invention will be further explained below with reference to the accompanying drawings;

[0031] like Figure 1 As shown in the figure, an implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization mainly includes high- and low-frequency feature enhancement, 2D-3D feature fusion, and iterative optical flow estimation. The specific steps are as follows:

[0032] Step 1. This embodiment uses the Flyingthing3D dataset and the VBOF (Variable Brightness Optical Flow) dataset as data sources. The Flyingthing3D dataset is a synthetic dataset designed specifically for optical flow estimation in 3D scenes. It contains image pairs of dynamic 3D objects and their corresponding real optical flow data. The images of the dynamic 3D objects are from flying objects simulated under different camera perspectives and lighting conditions. The VBOF dataset contains a set of images of the same scene under multiple exposure conditions. These images are taken by four different cameras and provide real optical flow data for each exposure condition.

[0033] Based on the normal-light RGB images and corresponding viewpoint depth maps provided by the Flyingthing3D dataset, the 3D point cloud corresponding to each pixel is calculated based on the depth map and camera intrinsic parameters. Simulated low-light noise is then added to the RGB image, along with an uncorrected white balance effect and noise model, and the image brightness is adjusted to generate synthetic low-light image data with low-light noise characteristics.

[0034] Step 2: For low-light image data, decompose it into high-frequency features and low-frequency features through the high- and low-frequency feature enhancement network, and enhance the original image using dual-frequency domain features. The specific steps are as follows:

[0035] s2.1. First, the low-light image pairs of the same scene are input into the convolution layer to obtain the low-light image features F. Then, the low-frequency features F of the low-light image are obtained from F through the average pooling operation. low .

[0036] For low-frequency features F low Perform bilinear interpolation upsampling and subtract the upsampling result from F to obtain the high-frequency feature F of the low-light image high .

[0037] s2.2, Since high-frequency information mainly represents the details of the image, using a smaller receptive field can better focus on local image information, thereby enhancing details more accurately. Therefore, the high-frequency feature F high Input into dense convolutional network:

[0038]

[0039] Dense (⋅) represents a dense convolutional network. A dense convolutional network consists of multiple small convolution kernels and has a residual connection structure. It can better focus on detailed areas and help explore high-frequency information.

[0040] Low-frequency information contains the background and contour of the image, which can be used to capture long-range dependencies. First, the low-frequency feature F lowPerform two consecutive downsamplings, and the feature obtained by each downsampling is F low1 、F low2 , the low-frequency features of the three scales are input into the channel self-attention respectively to capture the global background information and obtain the enhanced multi-scale low-frequency features F' low 、F' low1 、F' low2 , and finally fused through the Wavelet Fusion Network:

[0041]

[0042] Where WF(⋅) represents the wavelet fusion operation.

[0043] s2.3, high-frequency features after processing With low frequency characteristics Splicing is performed on the channel, and after one-dimensional convolution and channel attention mechanism, residual connection is performed with the low-light image feature F to generate the enhanced image feature F en The high- and low-frequency feature enhancement networks are able to process damaged image texture details and edge contours respectively, thereby improving feature extraction capabilities.

[0044] s2.4, enhance the image feature F en Input into the CNN network-based encoder to extract image features of low-light images and contextual features .

[0045] Step 3: Through the 2D-3D feature fusion network, the point cloud features are aligned and fused with the RGB features using pre-trained Encoder encoding and learnable interpolation projection, implicitly supervising the feature extraction of low-quality data.

[0046] s3.1. First, extract two-dimensional features from RGB image pairs of the same scene through pre-trained CNN , 、 Represent two-dimensional features The height and width, Indicates the number of channels. Extract 3D features from point cloud data ,At the same time, obtain the location information of the point cloud from the depth map, M represents the number of point clouds, Representing 3D features of point cloud data The number of channels.

[0047] s3.2. Since image features are dense and point cloud features are sparse, in order to achieve feature alignment, a learnable interpolation method is applied to point cloud features. Converted to dense point cloud features of the same size as image features For each image pixel, the neighborhood features are weighted by ScoreNet based on coordinate offset, and then The nearest neighbor method is used to find the projection points of the point cloud features:

[0048]

[0049] in, Represents the offset of each pixel about the 2D projection point of the point cloud. ScoreNet() represents the ScoreNet network. Represents KNN nearest neighbors. Representing dense point cloud features Features of pixel (i, j), i=1,2…H, j=1,2,…W.

[0050] s3.3, through Convolution on dense point cloud features and image features Fusion is performed in the channel dimension:

[0051] Conv

[0052] Normal image features after fusion and normal context features It is used to implicitly provide guidance to the high- and low-frequency feature enhancement network and optimize the feature extraction capability. Supervise the feature extraction process of the high and low frequency feature enhancement network:

[0053]

[0054] || ||2 means calculating the 2-norm.

[0055] Step 4: Construct a 4D correlation volume table and use GRU to infer optical flow.

[0056] s4.1. Image features based on low-light images and contextual features Calculate 4D correlation volume , used to establish feature correspondences between pixels.

[0057] s4.2, using multi-scale correlation volume, through the GRU-based update operator to iterate and gradually optimize the optical flow estimation. Assuming the initial optical flow field , in In iterations, update the optical flow estimate:

[0058]

[0059]

[0060] in, is the relevant feature retrieved from the relevant volume C, and GRU() represents the gated recurrent unit.

[0061] s4.3, by predicting optical flow With real optical flow between Distance is supervised and the prediction sequence is Apply exponential decay weights , calculate the optical flow estimation loss :

[0062]

[0063] in , Q represents the number of iterative updates of GRU.

[0064] In this embodiment, the exponential decay weight It is set to 0.9 and 12 optical flow prediction iterations are performed to achieve better coarse-to-fine optical flow updates.

[0065] The final loss function Set as prior feature loss and optical flow estimation loss sum:

[0066]

[0067] in, The model in this paper is implemented using PyTorch and trained using the Adam optimizer for a total of 200,000 iterations. The initial learning rate is set to , and gradually reduced to .

[0068] The original image from the Flyingthing3D dataset is denoted as C, and the processed image with low light noise features is denoted as CN.

[0069] First, a portion of the original images C is selected for model training, and the remaining original images are used as a test set for performance testing. Endpoint error (EPE) and 1 pixel accuracy (ACC 1px) are selected as evaluation indicators to compare the performance of this method with the existing technical methods. The test results are shown in Table 1:

[0070]

[0071] Table 1

[0072] As shown in Table 1, our method achieves an EPE of 2.91 and an ACC 1px of 86.54%, surpassing all other methods. Compared to the second-ranked GMFlow, our method improves EPE by approximately 9.3% and by approximately 19.3% compared to RAFT. Experimental results demonstrate that introducing implicit feature supervision during training indirectly guides the model to learn effective feature extraction, significantly improving subsequent optical flow estimation.

[0073] Then, the original image C and the image CN with low light noise features are used for model training, and the performance of different methods on the original normal image and the image after noise injection is compared, as shown in Table 2:

[0074]

[0075] Table 2

[0076] The data in Table 2 shows that our method achieves the best performance across all metrics. Compared to GMFlow, our method improves the EPE metric by 7.1% and 7.8% on normal images and noise-injected images, respectively. This demonstrates that, with the support of the high- and low-frequency feature enhancement network and prior feature guidance, our method outperforms models trained solely on RGB images in addressing the challenges of optical flow estimation.

[0077] In order to compare the performance of different models in low-light optical flow estimation, tests were conducted on the VBOF dataset, as shown in Table 3:

[0078]

[0079] Table 3

[0080] On the VBOF dataset and its SONY subset, our method achieved the best results on the SONY subset, with an EPE of 20.05, approximately 1.5% higher than the second-place GMFlow. Furthermore, our method also achieved the best results on the entire VBOF dataset, improving upon RAFT by approximately 1.1%. This demonstrates that our method, trained on synthetic low-light images, is effective in handling real-world low-light noisy scenes.

[0081] Figure 2The comparison of the visual results of optical flow estimation of different methods is shown. The first row shows the output results of different methods for clear RGB images. It can be seen that under the guidance of implicit features, this method can effectively track the contours of small objects and perform well in estimating the optical flow of moving objects. The second row shows the estimation results for RGB images injected with noise. Despite the addition of noise, this method still shows strong noise resistance, can accurately restore the information in the damaged image, and maintain a high optical flow estimation accuracy. The third row shows images in real low-light environments. Through training with synthetic data, this method helps to effectively handle optical flow estimation in the face of real environmental noise. In contrast, other methods perform poorly in areas with large noise and when the object contours are elongated, highlighting the limitations of image features affected by noise in the absence of additional prior guidance.

[0082] Finally, in order to verify the effectiveness of the high- and low-frequency feature enhancement network and prior feature loss in this method, a series of ablation experiments were conducted. The results are shown in Table 4:

[0083]

[0084] Table 4

[0085] As can be seen, the prior feature loss improves the performance of the RAFT framework by about 7.9%, while the high- and low-frequency feature enhancement network improves the performance by about 4.1%. Applying both the prior feature loss and the high- and low-frequency feature enhancement network together improves the performance by about 12.7%.

[0086] In summary, this application proposes a novel multimodal collaborative implicit image enhancement method, aiming to improve the performance of optical flow estimation in challenging environments. By introducing multimodal (RGBD) collaborative training, the quality of the input image is effectively improved, thereby achieving more accurate optical flow estimation under complex conditions. The collaborative learning of RGB and depth images at the feature level enables the enhancement network to implicitly acquire multimodal knowledge with geometric consistency, thereby improving the feature extraction capability of optical flow calculation. Extensive experimental verification has been carried out on synthetic and real datasets, and the results show that the performance of optical flow estimation can be improved by this method.

Claims

1. An implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization, characterized by: The specific steps include: Step 1: Obtain an RGB image under normal lighting conditions and a depth map of the corresponding viewing angle; calculate the corresponding 3D point cloud data based on the depth map and camera intrinsic parameters; Use RGB images in normal lighting scenes to synthesize low-light image data; Step 2: Build a high- and low-frequency feature enhancement network and input low-light images; First, high-frequency features and low-frequency features are decomposed; then, a dense convolutional network is used to enhance high-frequency features, and multi-scale attention enhancement and wavelet fusion are used to enhance low-frequency features; Then, after the channel attention mechanism and residual connection weighted fusion, the enhanced image feature F is generated. en ; Finally, the encoder is used to extract the enhanced image features F en Extracting image features from low-light images and contextual features ; Step 3: Build a 2D-3D feature fusion network to extract two-dimensional image features from RGB images and 3D point cloud features in 3D point clouds , after feature alignment and fusion, we get the normal lighting image features and contextual features ; Through the prior feature loss function Supervise the feature extraction process of the high and low frequency feature enhancement network: || ||2 means calculating the 2-norm; Step 4: Image features based on low-light images and contextual features Calculate 4D correlation volume , using multi-scale correlation volume, iteratively optimize the initial optical flow field through the recursive update operator of GRU, gradually refine the optical flow estimation results, and calculate the optical flow estimation loss ; Step 5. Set the total loss function is the prior feature loss and optical flow estimation loss The weighted sum of is used to complete the model training; the image pairs that need to be estimated for optical flow are input into the trained high- and low-frequency feature enhancement network to extract image features. and contextual features , and then output the optical flow estimation result through the method of step 4.

2. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization according to claim 1, characterized in that: Simulated low-light noise is added to RGB images in normal lighting scenes, uncorrected white balance effects and noise models are introduced, and image brightness is adjusted to synthesize low-light image data with low-light noise characteristics.

3. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization according to claim 1, characterized in that: The high- and low-frequency feature enhancement network decomposes the input low-light image into high-frequency features and low-frequency features, and enhances the original image using dual-frequency domain features. The specific steps are as follows: s2.

1. First, the low-light image is input into the convolution layer to extract the low-light image features F; then the low-frequency features F of the low-light image are obtained through the average pooling operation. low ; For low-frequency features F low Perform bilinear interpolation upsampling and subtract the upsampling result from F to obtain the high-frequency feature F of the low-light image high ; s2.2, the high frequency feature F high Input into the dense convolutional network to obtain enhanced high-frequency features ; For low-frequency features F low Perform two consecutive downsamplings, and the feature obtained by each downsampling is F low1 、F low2 , the low-frequency features of the three scales are input into the channel self-attention respectively to capture the global background information and obtain the enhanced multi-scale low-frequency features F' low 、F' low1 、F' low2 , and finally fused through the wavelet fusion network to obtain enhanced low-frequency features ; Where WF(⋅) represents the wavelet fusion operation; s2.3, yes and Splicing is performed on the channel, and after one-dimensional convolution and channel attention mechanism, residual connection is performed with the low-light image feature F to generate the enhanced image feature F en .

4. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization according to claim 1, characterized in that: The 2D-3D feature fusion network aligns the point cloud features with the RGB features and then fuses them. The specific steps are as follows: s3.

1. First, extract two-dimensional features from RGB images under normal lighting scenes through a pre-trained encoder ; Extract 3D features from point cloud data ,At the same time, obtain the location information of the point cloud from the depth map; s3.2, Apply the learnable interpolation method to transform the point cloud features Converted to dense point cloud features of the same size as image features ; s3.3, dense point cloud features and image features Fusion is performed in the channel dimension.

5. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization according to claim 4, characterized in that: For each image pixel, the neighborhood features are weighted by ScoreNet based on coordinate offset, and then The nearest neighbor method is used to find the projection points of the point cloud features: in, Represents the offset of each pixel about the 2D projection point of the point cloud; ScoreNet() represents the ScoreNet network; represents KNN nearest neighbor; Representing dense point cloud features The features of pixel (i, j), i=1,2…H, j=1,2,…W, 、 Represent two-dimensional features The height and width of the 6. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization according to any one of claims 1 or 4, characterized in that: An encoder based on a CNN network structure is used for feature extraction.

7. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization according to claim 1, characterized in that: The optical flow estimation loss : in represents exponentially decaying weights and , Q represents the number of iterative updates of GRU.

8. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization according to claim 1 or 7, characterized in that: The loss function for: in, represents the loss weight.

9. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization according to claim 8, characterized in that: Setting loss weights =0.2; use Adam optimizer for training, set the model iteration number to 200,000 times, and the initial learning rate to , and is reduced to during the training process through the cosine annealing strategy .

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 5 or 7.

Images