Implicit image enhancement and optical flow estimation method based on multi-mode collaborative optimization

Abstract

The invention discloses an implicit image enhancement and optical flow estimation method based on multi-mode collaborative optimization. The method comprises the following steps: firstly, acquiring an RGB image in a normal illumination scene and a depth map of a corresponding view angle; and calculating corresponding three-dimensional point cloud data according to the depth map and the camera internal reference. And synthesizing low-illumination image data by using the RGB images in the normal illumination scene. Decomposing high-frequency features and low-frequency features of the low-illumination image by using a high-low-frequency feature enhancement network to enhance the low-illumination image, and then extracting image features and context features of the low-illumination image; and extracting 2D image features and 3D point cloud features by using a 2D-3D feature fusion network, and performing alignment fusion to obtain image features and context features of a normal RGB image for supervising the feature extraction process of the high and low frequency feature enhancement network. And finally, based on the image features and context features of the low-illumination image, constructing a 4D correlation volume table, and reasoning the optical flow by using GRU.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to an implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization. Background Art

[0002] Optical flow estimation aims to calculate the pixel-level two-dimensional motion between consecutive frames and describe the motion field in two-dimensional space. Traditional optical flow algorithms usually rely on the photometric consistency assumption and derive motion information through optimization. The core of this type of method is to manually extract features, while the introduction of deep learning has significantly changed the way optical flow is estimated. Early deep learning optical flow methods (such as FlowNet) directly estimated optical flow from image pairs through two convolutional neural networks, eliminating the manual feature extraction step in traditional methods and verifying the feasibility of learning optical flow directly from images.

[0003] With the development of deep learning, the design of optical flow estimation networks has become more diverse. For example, PWC-Net adopts a spatial pyramid structure to gradually estimate optical flow at multiple levels, and introduces a feature pyramid and a matching method based on cost volume, which improves the accuracy of optical flow estimation while maintaining computational efficiency. RAFT uses a novel network structure to use multi-scale 4D correlation volume search and GRU iterative updates of optical flow, while GMA enhances global motion matching capabilities in feature extraction and encoding through global motion information aggregation and local attention mechanism, thereby improving the performance of optical flow calculation.

[0004] However, existing optical flow methods are mainly designed for image data collected under normal environmental conditions. In complex environments, image data is often affected by information loss. For example, in low-light conditions, increased noise and weakened texture features can significantly degrade image quality, destroying the photometric consistency assumption that optical flow estimation relies on. The increase in noise and the loss of texture details reduce the texture consistency between frames, resulting in a decrease in the performance of the subsequent optical flow estimation network.

[0005] For optical flow estimation in challenging environments such as severe weather conditions or low-light environments, corresponding solutions have also emerged in the prior art.

[0006] For example, in rainy scenes, RobustNet estimates optical flow by utilizing rain-free residual channels, while RainFlow reduces the impact of rain streaks and haze on image features by generating features that are invariant to rain streaks and rain fog. In haze scenes, some methods perform image dehazing and optical flow estimation simultaneously by synthesizing haze data and applying style transfer techniques.

[0007] In low-light environments, Zheng et al. proposed a synthetic dark noise optical flow benchmark dataset called FlyingChairs Dark&Noise (FCDN) dataset, which solves the problem of missing datasets by adding dark image noise to normal lighting datasets. They also introduced Various Brightness Optical Flow (VBOF) datasets with different exposure levels, which contains images with different exposure levels and optical flow pseudo-labels. In addition, CEDFlow introduces high- and low-frequency adaptive enhancement, as well as edge enhancement of feature dimensions, and specially designs a structure for image feature extraction in low-light environments to improve the performance of optical flow estimation.

[0008] However, extracting image features solely based on the limited information in low-light images limits the performance of optical flow estimation. The reduced light intake in low-light conditions makes it difficult to extract necessary information, which hinders the extraction of high-quality features that are critical for subsequent optical flow calculations. In addition, since the network is trained only using low-quality images, this reduces the performance that can usually be achieved under normal lighting conditions. Summary of the invention

[0009] In view of the shortcomings of the prior art, the present invention proposes an implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization. By combining RGB and depth image data and introducing multimodal information and a multimodal collaborative optimization framework, the robustness and accuracy of optical flow estimation are significantly improved, and the shortcomings of the prior art in the limited performance of optical flow estimation in difficult scenes such as low light, high noise or other complex dynamic scenes are solved. This method performs particularly well in scenes with low-quality images.

[0010] An implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization specifically includes the following steps: Step 1: Get the RGB image under normal lighting and the depth map of the corresponding viewing angle. Calculate the 3D point cloud corresponding to the pixel based on the depth map and camera intrinsic parameters.

[0011] Add simulated low-light noise to the RGB image and adjust the image brightness to synthesize low-light image data.

[0012] Step 2: Build a high- and low-frequency feature enhancement network. For the low-light image data obtained in step 1, decompose it into high-frequency features and low-frequency features. The high-frequency features contain detailed information of the image, such as edges and textures. The low-frequency features are the background and overall outline of the image.

[0013] The high-frequency features are input into the dense convolutional network for enhancement to highlight the texture details; the low-frequency features are processed by the multi-scale feature enhancement network, and the attention mechanism is used to capture the global background information. Then, the enhanced image features F are generated through the channel attention mechanism and residual connection weighted fusion. en .

[0014] Extracting image features of low-light image data from enhanced image features by an encoder and contextual features .

[0015] Step 3: Use the encoder to extract 2D image features from the RGB image and 3D point cloud under normal lighting scenes. and 3D point cloud features , after feature alignment and fusion, the fused normal illumination image features are obtained and contextual features .

[0016] Through the prior feature loss function Supervise the feature extraction process in step 2: || ||2 means calculating the 2-norm.

[0017] Step 4: Image features based on low-light images and contextual features Calculate 4D correlation volume , using multi-scale correlation volumes, using the recursive update operator of GRU (Gated Recurrent Unit) to iteratively optimize the initial optical flow field, gradually refine the optical flow estimation results, and calculate the optical flow estimation loss .

[0018] Step 5: Set the total loss function is the prior feature loss and optical flow estimation loss The weighted sum of completes the model training: in, represents the loss weight.

[0019] The image pairs that need to be estimated for optical flow are input into the trained high- and low-frequency feature enhancement network to extract image features. and contextual features , and then output the optical flow estimation result through the method in step 4.

[0020] The present invention has the following beneficial effects: 1. Use multimodal data to supplement the information missing of the original low-quality images and break through the performance bottleneck of the single-modal optical flow estimation method in complex scenes. Through the decomposition and enhancement of high- and low-frequency features, the image details and global information extraction capabilities are optimized respectively, and the accuracy of optical flow calculation is significantly improved. Introduce an implicit feature supervision mechanism, guide the training of the enhancement network through multimodal features, use RGBD fusion and 3D to 2D projection to ensure that the enhanced features maintain geometric interpretability, and achieve deep collaborative optimization of tasks and enhancements.

[0021] 2. This method is applicable to a wide range of computer vision tasks, including robot visual navigation, autonomous driving environment perception, moving object detection and tracking, etc., and performs particularly well in difficult scenarios such as low-light and high-noise environments. In addition, this technology also has potential value in image restoration, video analysis, and other application scenarios that require high-precision dynamic scene estimation. It provides a new solution for optical flow estimation in complex environments. Through implicit multimodal knowledge guidance and feature enhancement, it improves the performance of optical flow estimation in difficult scenarios, further promoting technological progress in the field of optical flow calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Flowchart of the implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization.

[0023] Figure 2 This is a qualitative comparison of different methods in the embodiments on the Flyingthing3D dataset. DETAILED DESCRIPTION

[0024] The present invention will be further explained below with reference to the accompanying drawings; like Figure 1 As shown in the figure, an implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization mainly includes high- and low-frequency feature enhancement, 2D-3D feature fusion, and iterative optical flow estimation. The specific steps are as follows:

[0025] Step 1. This embodiment uses the Flyingthing3D dataset and the VBOF (Variable Brightness Optical Flow) dataset as data sources. The Flyingthing3D dataset is a synthetic dataset designed for optical flow estimation in 3D scenes, which contains image pairs with dynamic 3D objects and their corresponding optical flow real data. The images of the dynamic 3D objects come from flying objects that simulate different camera perspectives and lighting conditions. The VBOF dataset contains a set of images of the same scene under multiple exposure conditions, which are taken by four different cameras and provide real optical flow data for each exposure condition.

[0026] Based on the normal lighting RGB image and the corresponding perspective depth map provided by the Flyingthing3D dataset, the 3D point cloud corresponding to the pixel is calculated according to the depth map and the camera intrinsic parameters. Then, simulated low-light noise is added to the RGB image, the uncorrected white balance effect and noise model are introduced, and the image brightness is adjusted to generate synthetic low-light image data with low-light noise characteristics.

[0027] Step 2: For low-light image data, the high- and low-frequency feature enhancement network is used to decompose it into high-frequency features and low-frequency features, and the dual-frequency domain features are used to enhance the original image. The specific steps are as follows: s2.1. First, the low-light image pairs of the same scene are input into the convolution layer to obtain the low-light image features F. Then, the low-frequency features F of the low-light image are obtained from F through the average pooling operation. low .

[0028] For low frequency features F low Perform bilinear interpolation upsampling, and then subtract the upsampling result from F to obtain the high-frequency feature F of the low-light image high .

[0029] s2.2. Since high-frequency information mainly represents the details of the image, a smaller receptive field can better focus on local image information, thereby enhancing the details more accurately. high Input into the dense convolutional network:

[0030] Dense (⋅) represents a dense convolutional network. A dense convolutional network includes multiple small convolution kernels and has a residual connection structure, which can better focus on detail areas and help explore high-frequency information.

[0031] Low-frequency information contains the background and contour of the image, which can be used to capture long-range dependencies. First, the low-frequency feature F low Perform two consecutive downsamplings, and the feature obtained by each downsampling is F low1 、F low2 , the low-frequency features of the three scales are input into the channel self-attention respectively to capture the global background information and obtain the enhanced multi-scale low-frequency features F' low 、F' low1 、F' low2 , and finally fused through the Wavelet Fusion Network:

[0032] Where WF(⋅) represents the wavelet fusion operation.

[0033] s2.3, high frequency features after processing With low frequency characteristics Splicing is performed on the channel, and after one-dimensional convolution and channel attention mechanism, residual connection is performed with the low-light image feature F to generate enhanced image feature F en The high- and low-frequency feature enhancement networks are able to process damaged image texture details and edge contours respectively, thereby improving feature extraction capabilities.

[0034] s2.4, the enhanced image feature F en Input into the CNN network-based encoder to extract image features of low-light images and contextual features .

[0035] Step 3: Through the 2D-3D feature fusion network, the point cloud features are aligned and fused with the RGB features using the pre-trained Encoder encoding and the learnable interpolation projection method, implicitly supervising the feature extraction of low-quality data.

[0036] s3.1. First, extract two-dimensional features from RGB image pairs of the same scene through pre-trained CNN , , Represents two-dimensional features The height and width of Indicates the number of channels. Extracting 3D features from point cloud data ,At the same time, obtain the location information of the point cloud from the depth map, M represents the number of point clouds, Representing 3D features of point cloud data The number of channels.

[0037] s3.2. Since image features are dense and point cloud features are sparse, in order to achieve feature alignment, a learnable interpolation method is used to interpolate point cloud features. Converted to dense point cloud features of the same size as image features For each image pixel, the neighborhood features are weighted by ScoreNet based on coordinate offset, and then The nearest neighbor method is used to find the projection point of the point cloud feature:

[0038] in, Represents the offset of each pixel about the 2D projection point of the point cloud. ScoreNet() represents the ScoreNet network. Represents KNN nearest neighbors. Representing dense point cloud features Features of pixel (i,j), i=1,2…H, j=1,2,…W.

[0039] s3.3, through Convolution on dense point cloud features and image features Fusion is performed on the channel dimension: Conv Normal image features after fusion and normal context features It is used to implicitly provide guidance to the high- and low-frequency feature enhancement network and optimize the feature extraction capability. Supervise the feature extraction process of the high and low frequency feature enhancement network:

[0040] || ||2 means calculating the 2-norm.

[0041] Step 4: Construct a 4D correlation volume table and use GRU to infer optical flow.

[0042] s4.1. Image features based on low-light images and contextual features Calculate 4D correlation volume , used to establish feature correspondence between pixels.

[0043] s4.2, using multi-scale correlation volumes, iterate through the GRU-based update operator to gradually optimize the optical flow estimation. Assume that the initial optical flow field , in In iterations, update the optical flow estimate:

[0044] in, is the relevant feature retrieved from the relevant volume C, and GRU() represents the gated recurrent unit.

[0045] s4.3, by predicting optical flow With real optical flow Between Distance is supervised and the prediction sequence is Apply exponential decay weights , calculate the optical flow estimation loss : in , Q represents the number of iterative updates of GRU.

[0046] In this embodiment, the exponential decay weight It is set to 0.9 and 12 optical flow prediction iterations are performed to achieve better coarse-to-fine optical flow updates.

[0047] The final loss function Set as prior feature loss and optical flow estimation loss sum: in, represents the loss weight, which is set to 0.2 in this example. The model in this article is implemented using PyTorch and trained using the Adam optimizer for a total of 200,000 iterations. The initial learning rate is set to , and gradually decreases to .

[0048] The original image from the Flyingthing3D dataset is denoted as C, and the processed image with low light noise features is denoted as CN.

[0049] First, select part of the original image C for model training, and use the remaining original images as the test set for performance testing. Select endpoint error (EPE) and 1 pixel accuracy (ACC 1px) as evaluation indicators to compare the performance of this method with the existing technology method. The test results are shown in Table 1: Table 1 From the data in Table 1, we can see that the EPE of this method is 2.91 and the ACC 1px is 86.54%, which exceeds all other methods. Compared with GMFlow, which ranks second, this method improves EPE by about 9.3% and by about 19.3% compared with RAFT. The experimental results show that the introduction of implicit feature supervision during the training process can indirectly guide the model to learn effective feature extraction, thereby significantly improving the subsequent optical flow estimation effect.

[0050] Then, the original image C and the image CN with low light noise features are used for model training, and the performance of different methods on the original normal image and the image after noise injection is compared, as shown in Table 2: Table 2 The data in Table 2 show that this method has achieved the best performance in all indicators. Compared with GMFlow, the EPE indicators of this method on normal images and injected noise images are improved by 7.1% and 7.8% respectively, indicating that with the support of high- and low-frequency feature enhancement networks and prior feature guidance, this method is better than the model trained only on RGB images in solving the challenge of optical flow estimation.

[0051] In order to compare the performance of different models in low-light optical flow estimation, tests were conducted on the VBOF dataset, as shown in Table 3: Table 3 On the VBOF dataset and its SONY subset, this method achieved the best result on the SONY subset, with an EPE of 20.05, which is about 1.5% higher than the second-place GMFlow. In addition, this method also achieved the best result on the entire VBOF dataset, which is about 1.1% higher than RAFT, indicating that this method can effectively cope with low-light noise scenes in the real world by training with synthetic low-light images.

[0052] Figure 2 The comparison of the visualization results of optical flow estimation of different methods is shown. The first row shows the output results of different methods for clear RGB images. It can be seen that under the guidance of implicit features, this method can effectively track the contours of small objects and perform well in estimating the optical flow of moving objects. The second row shows the estimation results for RGB images injected with noise. Despite the addition of noise, this method still shows strong noise resistance, can accurately restore the information in the damaged image, and maintain a high optical flow estimation accuracy. The third row shows images in real low-light environments. This method is trained with synthetic data, which helps to effectively handle optical flow estimation in the face of real environmental noise. In contrast, other methods perform poorly in areas with large noise and when the outlines of objects are elongated, highlighting the limitations of image features affected by noise in the absence of additional prior guidance.

[0053] Finally, in order to verify the effectiveness of the high- and low-frequency feature enhancement network and the prior feature loss in this method, a series of ablation experiments were conducted. The results are shown in Table 4: Table 4 It can be seen that the prior feature loss improves the performance of the RAFT framework by about 7.9%, while the high- and low-frequency feature enhancement network improves the performance by about 4.1%. Applying the prior feature loss and the high- and low-frequency feature enhancement network at the same time, that is, the present method improves the performance by about 12.7%.

[0054] In summary, this application proposes a novel multimodal collaborative implicit image enhancement method, which aims to improve the performance of optical flow estimation in challenging environments. By introducing multimodal (RGBD) collaborative training, the quality of the input image is effectively improved, thereby achieving more accurate optical flow estimation under complex conditions. The collaborative learning of RGB and depth images at the feature level enables the enhancement network to implicitly acquire multimodal knowledge with geometric consistency, thereby improving the feature extraction capability of optical flow calculation. A large number of experimental verifications have been carried out on synthetic and real datasets, and the results show that the performance of optical flow estimation can be improved by this method.

Claims

1. An implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization, characterized by: The specific steps include: Step 1: Obtain an RGB image under normal lighting and a depth map of the corresponding viewing angle; calculate the corresponding three-dimensional point cloud data based on the depth map and camera intrinsic parameters; Use RGB images in normal lighting scenes to synthesize low-light image data; Step 2: Build a high- and low-frequency feature enhancement network and input low-light images; First, high-frequency features and low-frequency features are decomposed; then, a dense convolutional network is used to enhance high-frequency features, and multi-scale attention enhancement and wavelet fusion are used to enhance low-frequency features; Then, after the channel attention mechanism and residual connection weighted fusion, the enhanced image feature F is generated. en ; Finally, the encoder is used to extract the enhanced image features F en Extracting image features from low-light images and contextual features ; Step 3: Build a 2D-3D feature fusion network to extract two-dimensional image features from RGB images and 3D point cloud features in 3D point clouds , after feature alignment and fusion, we get the normal illumination image features and contextual features ; Through the prior feature loss function Supervise the feature extraction process of the high and low frequency feature enhancement network: || ||2 means calculating the 2-norm; Step 4: Image features based on low-light images and contextual features Calculate 4D correlation volume , using multi-scale correlation volumes, the initial optical flow field is iteratively optimized through the recursive update operator of GRU, the optical flow estimation results are gradually refined, and the optical flow estimation loss is calculated ; Step 5: Set the total loss function is the prior feature loss and optical flow estimation loss The weighted sum of is used to complete the model training; the image pairs that need to be estimated by optical flow are input into the trained high- and low-frequency feature enhancement network to extract image features and contextual features , and then output the optical flow estimation result through the method in step 4.

2. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization as claimed in claim 1, characterized in that: Simulated low-light noise is added to RGB images in normal lighting scenes, uncorrected white balance effects and noise models are introduced, and image brightness is adjusted to synthesize low-light image data with low-light noise characteristics.

3. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization as claimed in claim 1, characterized in that: The high- and low-frequency feature enhancement network decomposes the input low-light image into high-frequency features and low-frequency features, and enhances the original image using dual-frequency domain features. The specific steps are as follows: s2.

1. First, the low-light image is input into the convolution layer to extract the low-light image feature F; then the low-frequency feature F of the low-light image is obtained through the average pooling operation. low ; For low frequency features F low Perform bilinear interpolation upsampling, and then subtract the upsampling result from F to obtain the high-frequency feature F of the low-light image high ; s2.2, high frequency features F high Input into the dense convolutional network to obtain enhanced high-frequency features ; For low frequency features F low Perform two consecutive downsamplings, and the feature obtained by each downsampling is F low1 、F low2 , the low-frequency features of the three scales are input into the channel self-attention respectively to capture the global background information and obtain the enhanced multi-scale low-frequency features F' low 、F' low1 、F' low2 Finally, the enhanced low-frequency features are obtained by fusion through the wavelet fusion network. ; Where WF(⋅) represents the wavelet fusion operation; s2.3, yes and Splicing is performed on the channel, and after one-dimensional convolution and channel attention mechanism, residual connection is performed with the low-light image feature F to generate enhanced image feature F en .

4. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization as claimed in claim 1, characterized in that: The 2D-3D feature fusion network aligns the point cloud features with the RGB features and then fuses them. The specific steps are as follows: s3.

1. First, extract two-dimensional features from RGB images under normal lighting scenes through a pre-trained encoder ; Extract 3D features from point cloud data ,At the same time, obtain the location information of the point cloud from the depth map; s3.2, Apply the learnable interpolation method to transform the point cloud features Converted to dense point cloud features of the same size as image features ; s3.3, for dense point cloud features and image features Fusion is performed on the channel dimension.

5. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization as claimed in claim 4, characterized in that: For each image pixel, the neighborhood features are weighted by ScoreNet based on coordinate offset, and then The nearest neighbor method is used to find the projection point of the point cloud feature: in, Represents the offset of each pixel about the 2D projection point of the point cloud; ScoreNet() represents the ScoreNet network; represents KNN nearest neighbor; Representing dense point cloud features Features of pixel (i,j), i=1,2…H, j=1,2,…W, , Represents two-dimensional features The height and width of the 6. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization according to any one of claims 1 or 4, characterized in that: The encoder based on CNN network structure is used for feature extraction.

7. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization as claimed in claim 1, characterized in that: The optical flow estimation loss : in represents exponential decay weight and , Q represents the number of iterative updates of GRU.

8. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization according to claim 1 or 7, characterized in that: The loss function for: in, represents the loss weight.

9. The implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization as claimed in claim 8, characterized in that: Setting loss weights =0.2; use Adam optimizer for training, set the model iteration number to 200,000 times, and the initial learning rate to , and is reduced to during the training process through the cosine annealing strategy .

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 5 or 7.

Images