A Dense Optical Flow Estimation Method and System Based on Multi-Plane Adaptive Rendering

Abstract

This invention proposes a dense optical flow estimation method and system based on multi-plane adaptive rendering, belonging to the field of computer vision technology. This invention constructs a dense optical flow estimation dataset containing realistic synthetic images, including high-quality datasets with real images and complex motion. During the training phase, the input data is first fed into the RAFT optical flow estimation network, and the loss function is calculated. Based on the calculated loss, the network updates its parameters using the gradient descent algorithm. During the usage phase, a model is created based on the saved model structure and parameters, and then dense optical flow estimation is performed on pairs of input images. This invention can efficiently synthesize high-quality real image data pairs and their optical flow labels without relying on any additional equipment, and the scenes of the data pairs can cover almost all scenes in daily life, ensuring the effectiveness and stability of this invention in dense optical flow estimation in real-world scenes.

Description

Technical Field

[0001] This invention relates to a dense optical flow estimation method and system, specifically to a method and system capable of acquiring a high-quality dense optical flow dataset containing real images, belonging to the field of computer vision technology. Background Technology

[0002] Optical flow refers to the movement relationship of pixels between two consecutive frames of an image. Optical flow data can be used in many computer vision applications, such as video compression, motion estimation, object tracking, and video analysis. Dense optical flow estimation is an important technical problem in the field of computer vision, and its purpose is to obtain the motion vector of each pixel in an image between two frames.

[0003] Optical flow estimation can be divided into sparse optical flow and dense optical flow, depending on whether sparse points in the image are selected for estimation. Sparse optical flow often selects some points with obvious features (large gradients) for estimation, while dense optical flow describes the optical flow of each pixel in the image moving to the next frame.

[0004] In recent years, with the development of deep learning, dense optical flow estimation methods based on deep learning have gradually become a research hotspot. Deep learning-based optical flow models are trained on paired image datasets with optical flow labels. Some methods use fluorescent markers to record motion in real-world scenes, but only sparse optical flow can be obtained. The KITTI dataset provides complex training data through sophisticated LiDAR and camera setups; however, the amount of data and the range of scenes in these real-world datasets are limited. Therefore, it is difficult to collect large amounts of realistic dense optical flow datasets in complex dynamic scenes, making it difficult to generalize models trained using deep supervised learning to more complex scenes. Currently, the common technique in this field is to train using training data containing a large number of realistic optical flow labels, with most images in the dataset obtained through computer simulation or animation. However, these large datasets alone are insufficient to train deep learning models for deployment in real-world environments because domain differences arise when transferring from synthetic images to real-world scenes.

[0005] In contrast, dense optical flow estimation methods based on real images have their own advantages because they are better adapted to various real-world scenes. These datasets typically consist of pairs of real images and synthesized new viewpoint images, with the optical flow between the two images serving as a label. Their performance largely depends on the realism of the synthesized new viewpoint images and optical flow labels based on real images. Regarding the input data, to synthesize realistic new viewpoint images and their corresponding optical flow labels, researchers use depth estimation models to construct a believable point cloud of the scene corresponding to the observed image. Then, under random motion vectors and rotation angles, the camera is virtually moved in the reconstructed environment, enabling the synthesis of a new view and corresponding optical flow labels. To pursue more realistic camera motion, recent methods directly use pre-estimated pseudo-optical flow labels and concatenate each pixel in the input image to the new viewpoint image. However, directly mapping pixels from the original real image to the new image introduces severe artifacts and holes, leading to a decrease in image realism, making it difficult to directly adapt this approach to real-world scenes. Neither of these techniques can synthesize high-quality new viewpoint images in real-world scenes. This can significantly degrade the performance of models trained on these datasets when dealing with real-world scenarios. Currently, there is no publicly available method for obtaining high-quality dense optical flow datasets based on real images. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and, in order to obtain high-quality, realistic image dense optical flow datasets, to creatively propose a dense optical flow estimation method and system based on multi-plane adaptive rendering. This invention can fill the gap in current dense optical flow data containing realistic images without requiring special equipment and with high efficiency.

[0007] The innovation of this invention lies in proposing a large-scale dataset for dense optical flow estimation in realistic dynamic scenes. High-resolution monocular images are collected from the internet, and scene depth is estimated using a monocular depth estimation model, thereby performing 3D reconstruction of the monocular images and constructing their multi-plane image representation. Then, using virtual camera poses and volume rendering technology, new viewpoint images corresponding to the camera poses are synthesized. Finally, a corresponding new viewpoint image and its corresponding optical flow label are synthesized for each image. Repeating the above process, a large-scale dense optical flow estimation dataset containing realistic synthesized images is constructed. This dataset includes high-quality data with realistic images and complex motion, compensating for the shortcomings of existing dense optical flow estimation datasets. Since it does not require the use of real cameras or video recording, the dataset can be collected more efficiently. In the training phase, the input data is first fed into the RAFT optical flow estimation model, and the loss function is calculated. Based on the calculated loss, the model updates its parameters using a gradient descent algorithm; after training, the model structure and parameters are saved. In the usage phase, a model is created based on the saved model structure and parameters, and then dense optical flow estimation is performed on pairs of images based on the input. The experimental results demonstrate the value of the dataset and the effectiveness of the proposed method across various metrics.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] Step 1: Collect real images containing actual motion and construct multi-plane image representations corresponding to the images. Synthesize images from new perspectives and dense optical flow labels using pixel mapping to construct the overall dataset.

[0010] Step 2: Train the model.

[0011] First, a feature encoding layer is applied to the model to extract low-level features from the two input images and the data is mapped to a high-dimensional space. Then, a feature association layer is used to generate four-dimensional association information for all pixels in the two images.

[0012] Then, a gated recurrent unit-based iterative update operator is used to find four-dimensional correlation information, thereby iteratively updating the predicted optical flow field. The loss value is calculated by comparing the dense optical flow labels and the predicted optical flow, and this loss value is fed back to the model parameters for training. Each neural model layer modifies its parameters based on the loss value fed back from the loss function, completing one training iteration.

[0013] Step 3: Use the trained model to estimate optical flow and generate the final predicted optical flow.

[0014] On the other hand, this invention proposes a dense optical flow estimation system based on multi-plane adaptive rendering, including a data collection module, a model training module, and an optical flow estimation module.

[0015] The data collection module is responsible for processing the video stream acquired by the camera in real time, extracting adjacent frames for image preprocessing, and synthesizing an optical flow dataset. The model training module uses the aforementioned optical flow dataset to train an optical flow estimation model. The optical flow estimation module is responsible for using the model trained by the model training module to execute the inference process and obtain the estimated dense optical flow.

[0016] The output of the data collection module is connected to the input of the model training module. The output of the model training module is connected to the input of the optical flow estimation module.

[0017] Beneficial effects

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] 1. This invention collects a large number of images from the Internet, synthesizes new images from new perspectives using multi-planar image representation, and obtains dense optical flow labels between the original and new images, thus directly acquiring optical flow datasets from real-world images. With a meticulously designed program, high-quality paired images and optical flow labels can be obtained through precise virtual camera poses. These include real images with realistic motion, which can efficiently supplement the dense optical flow dataset.

[0020] 2. This invention proposes a novel dense optical flow prediction method, which utilizes the aforementioned synthetic data containing real images for training, resulting in higher accuracy in optical flow prediction in real-world scenarios.

[0021] 3. This invention can efficiently synthesize high-quality real image data pairs and their optical flow labels without relying on any additional equipment, and the scenes of the data pairs can cover almost all scenes in daily life, which can ensure the effectiveness and stability of this invention in dense optical flow estimation in real scenes.

[0022] 4. This invention has high detection quality and is applicable to various devices such as mobile phone camera sensors and vehicle camera sensors. It is suitable for various application fields such as motion estimation and target tracking under real-world conditions. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method of the present invention.

[0024] Figure 2 This is a detailed schematic diagram of the high-quality real-image dense optical flow dataset of the present invention.

[0025] Figure 3 This is a schematic diagram illustrating the use and result evaluation of the dense optical flow estimation method of the present invention. Detailed Implementation

[0026] To better illustrate the purpose and advantages of the present invention, the present invention will be further described below with reference to the accompanying drawings.

[0027] like Figure 1 As shown, a dense optical flow estimation method based on multi-plane adaptive rendering includes the following steps:

[0028] Step 1: Data collection.

[0029] Step 1.1: Collect monocular images containing moving scenes.

[0030] The motion scenes include indoor scenes, outdoor scenes, and other scenes, ensuring that the image content of the scenes contained is close enough to the real scene.

[0031] Step 1.2: Input the collected images into the image synthesis module.

[0032] Specifically, the real image I s Defined as a matrix of size H×W×3, where H and W are the height and width of the image, respectively.

[0033] To construct a multi-plane image of the input image, monocular depth estimation is first used to obtain the estimated coarse depth and the RGBD image. Then, based on the contextual information of the RGBD image input, a novel plane adjustment model is designed to adjust the plane depths from the initial configuration, constructing a multi-plane image (MPI) of scene-specific depth, where each plane n corresponds to a depth d. n The image below, where each pixel on the plane contains the corresponding RGB color value c extracted by a neural model. n and volume density σ n .

[0034] Step 1.3: Let the original viewpoint be μ. s A new perspective for μ t To generate highly realistic images from the new perspective, a weighted sum of colors along a pixel ray is calculated from the new perspective, thus mapping a pixel from the old image to the new image. t The color corresponding to the new pixel.

[0035] Specifically, each pixel p1 on plane n of the new perspective image is projected onto pixel p0 on plane n of the original image through the camera intrinsic and extrinsic parameter matrices:

[0036] p1~KT 0→1 D0K -1 p0

[0037] Among them, T 0→1Let K be the camera transfer matrix from the original viewpoint to the target viewpoint; K be the camera's intrinsic parameter matrix; and D0 be the depth value. The optical flow (p1-p0) of each plane is calculated using pixel mapping relationships. Therefore, the color of each new plane can be obtained through bilinear sampling based on the mapped pixels.

[0038] To obtain the color of each plane in the new view image, I is calculated using the intersection points between the new plane and the rays passing through the scene. t :

[0039]

[0040] Among them, c n This represents the color obtained from bilinear interpolation on a new perspective image. This indicates whether the point lies on plane n, where N represents the number of planes in the multi-plane system.

[0041] Step 2: Model training.

[0042] The model parameters are trained using the real image dense optical flow dataset obtained in step 1.

[0043] Step 2.1: Input paired images (I t I s The feature encoding layer is defined as a matrix of size H×W×3, where H and W are the height and width of the image, respectively. To extract features from the pixels in the image, convolution and activation functions are first applied in the feature encoding layer to extract low-level features for each frame. These features are then stacked together to obtain the high-dimensional features of the entire sequence.

[0044] Step 2.2: After feature extraction of the input data, a feature association layer is used to generate H×W×H×W four-dimensional association information for all pixels, and a pooling layer is used to generate association information with a lower resolution.

[0045] Then, a cyclic update operator based on gated cyclic units is used to find four-dimensional correlation information. After that, the optical flow field is iteratively updated, with an initial value of 0.

[0046] Step 2.3: Calculate the iterative optical flow field and the dense optical flow label F s→t The loss is used to update the parameters of each layer through the backpropagation algorithm.

[0047] Step 3: Use the trained model to perform dense optical flow estimation and generate the final predicted optical flow.

[0048] Step 3.1: Acquire real video and adjacent frames using the camera.

[0049] In real-world motion scenarios, including indoor, outdoor, and other scenarios, videos are captured and the raw video input is obtained. The videos are then processed into multi-frame images, with each pair of adjacent frames serving as the model input.

[0050] Step 3.2: Transfer the input paired images (I t I s The pixels in the sequence are extracted as features, and these features are stacked together to obtain the high-dimensional features of the entire sequence.

[0051] Step 3.3: After feature extraction from the input data, a feature association layer is used to generate H×W×H×W four-dimensional association information for all pixels, and a pooling layer is used to generate association information at a lower resolution. Then, a cyclic update operator based on gated recurrent units is used to find the four-dimensional association information. Afterwards, the optical flow field is iteratively updated. The final iterative optical flow field is the estimated dense optical flow.

[0052] To achieve the objectives described in this invention, this invention further proposes a dense optical flow estimation system based on multi-plane adaptive rendering, including a data collection module and an optical flow estimation module.

[0053] The data collection module is responsible for collecting dense optical flow training datasets containing real images. The obtained data will be used in the model to help learn and update the corresponding model parameters.

[0054] In the optical flow estimation module, the RAFT model trained in the data collection module is selected. First, the input data is fed into the RAFT optical flow estimation model. A feature encoding layer is applied to the model to extract low-level features from the two input images and map the data to a high-dimensional space. Then, a feature association layer is used to generate four-dimensional association information for all pixels in the two images. Afterward, a gated recurrent unit-based iterative update operator is used to find the four-dimensional association information, thereby iteratively updating the predicted optical flow field.

[0055] The data collection module first synthesizes dense optical flow data containing real images, then inputs the data into the model for training, providing the original images, new perspective images, and dense optical flow labels. After training, the trained model is input into the optical flow estimation module and applied to dense optical flow prediction tasks in real-world scenes, demonstrating good performance and robustness under various motion, lighting conditions, and scene conditions.

[0056] To illustrate the effectiveness of the present invention, this embodiment will compare multiple methods under the same experimental conditions.

[0057] Table 1 compares other dense optical flow estimation schemes against the Sintel benchmark.

[0058]

[0059] Table 2 compares other dense optical flow estimation schemes against the KITTI test benchmark.

[0060]

[0061] As can be seen from the results in Tables 1 and 2, the method disclosed in this invention achieves excellent dense optical flow prediction results, both on the simulated Sintel dataset and on the KITTI dataset containing real test images. The evaluation results for mean endpoint error (EPE) and two error rates are reported here: the percentage of pixels with an absolute error greater than 3 (>3) or an absolute error and a relative error greater than 3% and 5% (Fl), respectively. These three metrics are widely used model evaluation metrics in video denoising tasks. Compared with other methods, this invention achieves better results on all three metrics. The results in Tables 1 and 2 demonstrate the superiority of this invention for dense optical flow estimation tasks on various datasets, significantly outperforming other methods.

[0062] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dense optical flow estimation method based on multi-plane adaptive rendering, characterized in that, Includes the following steps: Step 1: Collect real images containing real motion and construct multi-plane image representations corresponding to the images; use pixel mapping to synthesize images from new perspectives and dense optical flow labels to construct the overall dataset; Step 1.1: Collect monocular images containing moving scenes; The motion scenes include indoor scenes, outdoor scenes, and other scenes, ensuring that the image content of the scenes is close enough to the real scene. Step 1.2: Perform image synthesis on the collected images; Real images Defined as size The matrix, and These are the height and width of the image, respectively. First, monocular depth estimation is used to obtain the estimated coarse depth of the image and the RGBD image. Then, based on the contextual information of the RGBD image input, a plane adjustment model is designed to adjust the plane depth from the initial configuration, constructing a multi-plane image of the depth of a specific scene, where each plane... Corresponding to a depth The image below contains a portion of the image, and each pixel on the plane contains a corresponding RGB color value extracted by a neural model. and volume density ; Step 1.3: Let the original viewpoint be... A new perspective ; Calculate the weighted sum of colors along the pixel ray from a new perspective, mapping a pixel from the old image to the new image. The color corresponding to the pixel; A new perspective on the plane Each pixel on Both are projected onto the plane of the original image through the camera's intrinsic and extrinsic parameter matrices. pixels on : in, This is the camera transfer matrix from the original viewpoint to the target viewpoint; This is the camera's inherent intrinsic parameter matrix; The depth value is used, and the optical flow of each plane is calculated using pixel mapping relationships, i.e. ; To obtain the color of each plane in the new view image, the intersection points between the new plane and the rays passing through the scene are used to calculate... : in, This represents the color obtained from bilinear interpolation on a new perspective image. This indicates whether the point lies on plane n, where N represents the number of planes in the multi-plane system. Step 2: Train the model; First, a feature encoding layer is applied to the model to extract low-level features from the two input images, and the data is mapped to a high-dimensional space. Then, a feature association layer is used to generate four-dimensional association information for all pixels in the two images. Then, a gated recurrent unit-based loop update operator is used to find four-dimensional correlation information, thereby iteratively updating the predicted optical flow field; the comparison between dense optical flow labels and predicted optical flow is read to calculate the loss value, and the loss value is fed back to the model parameters for training; each neural model layer modifies its parameters according to the loss value fed back by the loss function, completing one training iteration; Step 2.1: Input paired images Defined as size The matrix, where, and These represent the height and width of the image, respectively; first, convolution and activation functions are applied in the feature encoding layer to extract low-level features for each frame; these features are then stacked together to obtain the high-dimensional features of the entire sequence; Step 2.2: After feature extraction from the input data, a feature association layer is used to generate features for all pixels. The four-dimensional association information is obtained, and a pooling layer is used to generate association information at a lower resolution. Then, a gated loop update operator is used to find four-dimensional correlation information; after that, the optical flow field is iteratively updated, with the initial value of the optical flow field being 0. Step 2.3: Calculate the iterative optical flow field and dense optical flow tags The loss is used to update the parameters of each layer through the backpropagation algorithm; Step 3: Use the trained model to estimate optical flow and generate the final predicted optical flow.

2. The dense optical flow estimation method based on multi-plane adaptive rendering as described in claim 1, characterized in that, Step 3 includes the following steps: Step 3.1: Acquire real video and adjacent frames using the camera; In real-world motion scenarios, including indoor, outdoor, and other scenarios, videos are captured and the raw video input is obtained. The videos are then processed into multi-frame images, with each pair of adjacent frames serving as the model input. Step 3.2: Input paired images The pixels in the sequence are extracted as features, and these features are stacked together to obtain the high-dimensional features of the entire sequence. Step 3.3: After feature extraction from the input data, a feature association layer is used to generate features for all pixels. The four-dimensional correlation information is obtained, and a pooling layer is used to generate lower-resolution correlation information. Then, a cyclic update operator based on gated cyclic units is used to find the four-dimensional correlation information. After that, the optical flow field is iteratively updated. The final iterative optical flow field is the estimated dense optical flow.

3. A dense optical flow estimation system based on multi-plane adaptive rendering that implements the method of claim 1, characterized in that, It includes a data collection module, a model training module, and an optical flow estimation module; The data collection module is responsible for processing the video stream acquired by the camera in real time, extracting adjacent frames for image preprocessing, and synthesizing an optical flow dataset; the model training module uses the aforementioned optical flow dataset to train an optical flow estimation model; and the optical flow estimation module is responsible for using the model trained by the model training module to execute the inference process and obtain the estimated dense optical flow. The output of the data collection module is connected to the input of the model training module; the output of the model training module is connected to the input of the optical flow estimation module.

Images