Optical Flow Estimation Method and Apparatus Based on Hierarchical Total Correlation Cost Volume Aggregation

Abstract

This invention proposes an optical flow estimation method based on hierarchical fully correlated cost volume aggregation. A feature encoding module extracts features from two frames of images and constructs a fully correlated cost volume to model the similarity between all pixels in the two images. The hierarchical fully correlated cost volume aggregation module first divides the fully correlated cost volume into smaller blocks and spatially recombines them to allow for cost aggregation using 2D and 3D convolutions. This module performs aggregation hierarchically, performing 2D convolutions within blocks for local matching aggregation and 3D convolutions between blocks for global matching aggregation. The optical flow iterative optimization module searches for matching information from the fully correlated cost volume based on the current estimated optical flow in each iteration. This information, along with the context features of the first frame image, is input into a convolutional GRU-based update model to obtain the residual of the current optical flow. After multiple iterations, the optical flow estimation result is obtained. This invention also provides a corresponding optical flow estimation device based on hierarchical fully correlated cost volume aggregation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision optical flow estimation technology, and more specifically relates to an optical flow estimation method and apparatus based on hierarchical fully correlated cost volume aggregation. Background Technology

[0002] Optical flow estimation is used to determine the direction and velocity of motion of objects in an image. It is a fundamental task in computer vision and plays a crucial role in various applications such as motion analysis, object tracking, and video interpolation.

[0003] Traditional methods typically define optical flow estimation as an energy minimization problem constrained by brightness constrained and spatially smooth. However, these methods often exhibit poor performance and limited generality. The rapid development of deep learning has led to significant progress in optical flow estimation. RAFT, one of the most prominent deep learning-based methods recently, computes pairwise correlations to generate a cost volume and leverages an iterative optimization framework to infer and refine pixel-level motion from the cost volume. RAFT has greatly improved the accuracy of optical flow estimation, establishing a new paradigm for the design of optical flow estimation networks. However, the cost volume constructed from simple feature correlations has limitations in encoding prior knowledge and non-local knowledge, which are crucial for accurately understanding complex scenes. Therefore, this method may produce false matches in regions with poor or ambiguous constraints, such as occluded and textureless regions (e.g.,...). Figure 1 (As shown). Although subsequent iterative optimization using contextual features as supplementary optimization can alleviate this problem to some extent, multiple iterations are still required. A common technique for solving this problem in stereo matching is to regularize the cost body through 2D or 3D convolution, i.e., cost aggregation. However, due to the high dimensionality of the fully correlated cost body, i.e., [H,W,H,W], it is not possible to directly aggregate the 4D cost body using 2D or 3D convolution.

[0004] Recent studies have attempted to achieve cost aggregation in optical flow by reducing the size of the 4D cost volume. For example, SeparableFlow separates the 2D motion of optical flow into two independent 1D problems: horizontal and vertical motion. Therefore, it compresses the 4D cost volume into two smaller 3D cost volumes and then aggregates the two separate cost volumes using a nonlocal aggregation layer. However, decomposing the 4D cost volume into horizontal and vertical components can lead to information loss and a decrease in accuracy. In contrast, Flowformer downsamples the 4D cost volume to reduce its size and uses self-attention for cost aggregation. Benefiting from the transformer's ability to capture global information, Flowformer achieves state-of-the-art performance. However, aggregation via self-attention incurs significant computational overhead. Furthermore, Flowformer requires specific retrieval methods due to the dimensionality changes of the original cost volume.

[0005] In summary, existing high-precision optical flow estimation methods suffer from long inference times and high memory usage when aggregating fully correlated cost volumes, which hinders the practical application of optical flow estimation algorithms. Summary of the Invention

[0006] The problem this invention aims to solve is to achieve high accuracy in optical flow estimation tasks by aggregating fully correlated cost volumes while reducing the number of model parameters, inference time, and GPU memory usage.

[0007] To achieve the above objectives, according to one aspect of the present invention, an optical flow estimation method based on hierarchical full correlation cost volume aggregation is proposed, comprising the following steps:

[0008] (1) Take two adjacent frames in the time dimension and call them the first frame and the second frame, respectively, to calculate the optical flow between the first frame and the second frame; extract the matching features of the two frames and the context features of the first frame;

[0009] (2) Select the feature map of the target size to construct the fully correlated cost body through matrix multiplication. The fully correlated cost body is spatially divided and reorganized to allow cost aggregation using 2D and 3D convolution. Then, hierarchical aggregation is performed through a local aggregation network based on 2D convolution and a global aggregation network based on 3D convolution. The aggregated cost body is used to construct the cost body pyramid.

[0010] (3) Based on the current estimated optical flow, multi-scale matching information is searched from the cost volume pyramid. After encoding with the context features of the first frame image, the optical flow residual is updated using a convolutional GRU-based update model. After multiple iterations, the final optical flow estimation result is obtained.

[0011] (4) Upsample the optical flow obtained in the last iteration to restore it to the original resolution, output the full-resolution optical flow map to obtain the optical flow estimate from the first frame image to the second frame image, and calculate the L1 loss between the optical flow estimate and the label to supervise the network.

[0012] In one embodiment of the present invention, step (1), extracting the matching features of the two frames and the contextual features of the first frame specifically includes:

[0013] The feature encoding module includes two sub-modules: a matching feature extraction sub-module and a context feature extraction sub-module. Both have the same structure and are composed of transformer-based networks. For two frames I1 and I2, the matching feature extraction sub-module extracts matching features f1, f2 ∈ R from the two frames respectively. D×H×W Where D is the dimension of the feature, and H and W are the height and width, respectively, which are 1 / 8 times the resolution of the original image; the context feature extraction submodule extracts the context features f from the first frame image I1. c ∈R D×H×W .

[0014] In one embodiment of the present invention, step (2) specifically includes:

[0015] The hierarchical fully correlated cost volume aggregation module performs hierarchical aggregation of the fully correlated cost volume. After extracting the matching features f1 and f2 of two frames of images, the fully correlated cost volume C is constructed by multiplying each pair of pixels point by point. Its dimensions are [H1, W1, H2, W2], where dimensions H1 and W1 are the height and width, respectively, and H2 and W2 represent the displacement range in the vertical and horizontal directions, respectively.

[0016] In one embodiment of the present invention, the hierarchical fully correlated cost volume aggregation module performs hierarchical aggregation of the fully correlated cost volume, specifically including:

[0017] The hierarchical fully correlated cost volume aggregation module divides aggregation into local aggregation and global aggregation. First, the fully correlated cost volume is divided into blocks and reorganized to allow aggregation using 2D and 3D convolutions. The fully correlated cost volume C is divided into N P×P blocks in the displacement range dimension [H2, W2]. The fully correlated cost volume is reorganized into a local cost volume C with dimensions [N, P × P, H1, W1]. local The local aggregation submodule applies 2D convolutions to the local cost volume C. local Cost aggregation yields a regularized cost body. Further through 2D convolutional layers The dimensions become [1, N, H1, W1]. The global aggregation submodule performs global aggregation on this cost volume through 3D convolution to obtain a regularized cost volume. Its dimensions are Further through the The cost volume is obtained by performing average pooling along the displacement range dimension. and The dimensions are respectively and Ultimately, a level 4 cost body pyramid was obtained. Provide displacement information of different magnitudes for the optical flow iterative optimization module.

[0018] In one embodiment of the present invention, the local aggregation submodule consists of a 1×1 2D convolution and an 11×11 large kernel depthwise separation convolution, with GELU as the activation function, and the global aggregation submodule consists of 4 layers of 3D convolution.

[0019] In one embodiment of the present invention, step (3) specifically includes:

[0020] In each iteration, the optical flow iterative optimization module searches for matching information corr from the cost volume pyramid based on the currently estimated optical flow flow, and compares it with the context features f of the first frame image. c The input is fed into the convolutional GRU-based update model, which outputs the hidden state h. The optical flow residual Δflow is obtained from the hidden state h through convolution. The optical flow residual is added to the current optical flow as the initial value of the optical flow for the next iteration. After multiple iterations, the optical flow estimation result is obtained.

[0021] In one embodiment of the present invention, the upsampling in step (4) specifically includes:

[0022] At each iteration, upsampling weights W are generated from the hidden state h through convolution. Based on the upsampling weights, a full-resolution optical flow graph is output by weighted combination of the neighbors of the low-resolution optical flow graph.

[0023] In one embodiment of the present invention, in step (4), the network is supervised by calculating the L1 loss between the optical flow estimate and the label, specifically including:

[0024] For the optical flow sequence {f1,...,f} output after N iterations, N Given an optical flow label f gt Supervision is performed using the L1 loss function, which is defined as follows:

[0025]

[0026] Where γ = 0.8, ||*|| represents taking the absolute value of *, and N is the total number of iterations.

[0027] In one embodiment of the present invention, in step (3), the optical flow value is set to 0 at the beginning of the iteration.

[0028] According to another aspect of the present invention, an optical flow estimation apparatus based on hierarchical fully correlated cost volume aggregation is also provided, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, wherein the instructions, after being executed by the processor, are used to complete the optical flow estimation method based on hierarchical fully correlated cost volume aggregation.

[0029] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0030] This invention segments and reassembles the fully relevant cost volume, using a hierarchical approach to perform local and global aggregation through convolution. Compared to Flowformer, a method based on self-attention aggregation, this invention achieves comparable accuracy on the Sintel Clean and KITTI 2015 benchmarks while reducing model parameters, inference time, and memory usage by 24.1%, 35.5%, and 21.6%, respectively. Attached Figure Description

[0031] Figure 1 This is a comparison diagram of the cost body and output optical flow of the method of the present invention and the cost body and output optical flow of RAFT.

[0032] Figure 2 This is a schematic diagram illustrating the principle of the optical flow estimation method based on hierarchical full correlation cost volume aggregation of the present invention;

[0033] Figure 3 This is a schematic diagram of the structure of the local aggregation module of the method of the present invention;

[0034] Figure 4 This is a comparison of the qualitative results of the method of this invention with other methods on the Sintel benchmark;

[0035] Figure 5 This is a comparison of the qualitative results of the method of this invention and other methods on the KITTI benchmark. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0037] To address the problems of existing technologies, this invention proposes an optical flow estimation method based on hierarchical fully correlated cost volume aggregation. This method utilizes a feature encoding module, a hierarchical fully correlated cost volume aggregation module, and an optical flow iterative optimization module. Specifically: the feature encoding module extracts features from two frames of images to construct a fully correlated cost volume, modeling the similarity between all pixels in the two images. The hierarchical fully correlated cost volume aggregation module first divides the fully correlated cost volume into smaller blocks and spatially recombines them to allow for cost aggregation using 2D and 3D convolutions. This module performs aggregation hierarchically, performing 2D convolutions within blocks for local matching aggregation and 3D convolutions between blocks for global matching aggregation. The optical flow iterative optimization module estimates the optical flow residual in each iteration, obtaining the optical flow estimation result after multiple iterations.

[0038] like Figure 2 As shown, this invention provides an optical flow estimation method based on hierarchical total correlation cost volume aggregation, comprising the following steps:

[0039] (1) Take two adjacent frames in the time dimension and call them the first frame and the second frame, respectively, to calculate the optical flow between the first frame and the second frame; extract the matching features of the two frames and the context features of the first frame;

[0040] In step (1), extracting the matching features of the two frames and the contextual features of the first frame specifically includes:

[0041] The feature encoding module includes two sub-modules: a matching feature extraction sub-module and a context feature extraction sub-module. Both have the same structure and are composed of transformer-based networks. For two frames I1 and I2, the matching feature extraction sub-module extracts matching features f1, f2 ∈ R from the two frames respectively. D×H×W Where D is the dimension of the feature, and H and W are the height and width, respectively, which are 1 / 8 times the resolution of the original image; the context feature extraction submodule extracts the context features f from the first frame image I1. c ∈R D×H×W .

[0042] (2) Select the feature map of the target size to construct the fully correlated cost body through matrix multiplication. The fully correlated cost body is spatially divided and reorganized to allow cost aggregation using 2D and 3D convolution. Then, hierarchical aggregation is performed through a local aggregation network based on 2D convolution and a global aggregation network based on 3D convolution. The aggregated cost body is used to construct the cost body pyramid.

[0043] Step (2) specifically includes: the hierarchical full correlation cost volume aggregation module performs hierarchical aggregation of the full correlation cost volume. After extracting the matching features f1 and f2 of the two frames of images, the full correlation cost volume C is constructed by multiplying each pair of pixels point by point. Its dimensions are [H1, W1, H2, W2], where dimensions H1 and W1 are the height and width, respectively, and H2 and W2 represent the displacement range in the vertical and horizontal directions, respectively.

[0044] Furthermore, the hierarchical fully correlated cost volume aggregation module performs hierarchical aggregation of the fully correlated cost volume, specifically including: the hierarchical fully correlated cost volume aggregation module divides the aggregation into local aggregation and global aggregation. First, the fully correlated cost volume is divided into blocks and reorganized to allow the use of 2D convolution and 3D convolution to aggregate the cost volume; the fully correlated cost volume C is divided into N blocks of size P×P in the displacement range dimension [H2, W2], where The fully correlated cost volume is reorganized into a local cost volume C with dimensions [N, P × P, H1, W1]. local The local aggregation submodule applies 2D convolutions to the local cost volume C. local Cost aggregation yields a regularized cost body. Further through 2D convolutional layers The dimensions become [1, N, H1, W1]. The global aggregation submodule performs global aggregation on this cost volume through 3D convolution to obtain a regularized cost volume. Its dimensions are Further through the The cost volume is obtained by performing average pooling along the displacement range dimension. and The dimensions are respectively and Ultimately, a level 4 cost body pyramid was obtained. This provides displacement information of different magnitudes for the optical flow iterative optimization module. For example, Figure 3 As shown, the local aggregation submodule consists of 1×1 2D convolutions and 11×11 large kernel depth separation convolutions, with GELU as the activation function, and the global aggregation submodule consists of 4 layers of 3D convolutions.

[0045] (3) Based on the current estimated optical flow, multi-scale matching information is searched from the cost volume pyramid. After encoding with the context features of the first frame image, the optical flow residual is updated using a convolutional GRU-based update model. After multiple iterations, the final optical flow estimation result is obtained.

[0046] Step (3) specifically includes: In each iteration, the optical flow iterative optimization module searches for matching information corr from the cost volume pyramid based on the current estimated optical flow flow, and compares it with the context features f of the first frame image. cThe input is fed into the convolutional GRU-based update model, which outputs the hidden state h. The optical flow residual Δflow is obtained from the hidden state h through convolution. The optical flow residual is added to the current optical flow as the initial value of the optical flow for the next iteration. After multiple iterations, the optical flow estimation result is obtained.

[0047] In step (3), the optical flow value is set to 0 at the beginning of the iteration.

[0048] (4) Upsample the optical flow obtained in the last iteration to restore it to the original resolution, output the full-resolution optical flow map to obtain the optical flow estimate from the first frame image to the second frame image, and calculate the L1 loss between the optical flow estimate and the label to supervise the network.

[0049] The upsampling in step (4) specifically includes: generating upsampling weights W from the hidden state h through convolution at each iteration, and outputting a full-resolution optical flow map by weighted combination of the neighbors of the low-resolution optical flow map according to the upsampling weights.

[0050] In step (4), the L1 loss between the optical flow estimate and the label is calculated to supervise the network, specifically including:

[0051] For the optical flow sequence {f1,...,f} output after N iterations, N Given an optical flow label f gy Supervision is performed using the L1 loss function, which is defined as follows:

[0052]

[0053] Where γ = 0.8, ||*|| represents taking the absolute value of *, and N is the total number of iterations.

[0054] Table 1 shows a comparison between the method of this invention and Flowformer, the current state-of-the-art method for aggregating cost volumes based on self-attention, in the embodiments of this invention. Inference time and memory usage were tested on a hardware platform equipped with an NVIDIA RTX 3090. The method of this invention achieves comparable accuracy on the Sintel Clean benchmark and the KITTI 2015 benchmark while reducing model parameters, inference time, and memory usage by 24.1%, 35.5%, and 21.6%, respectively.

[0055]

[0056] Table 1

[0057] Table 2 shows a comparison of the method of this invention with other optical flow estimation methods in the embodiments of this invention. The performance of this method is comparable to the state-of-the-art method FlowFormer. Furthermore, the performance of this method is superior to SeparableFlow and KPAFlow, but slightly lower on KITTI. This is because this method uses a transformer-based feature extraction network, and the 200 training images of KITTI-15 are insufficient to train a high-performance feature extraction network. Therefore, this method is fine-tuned using KITTI12 (Our*). After adding KITTI-12 data, the F1-all of this method further decreased from 4.67% to 4.24%.

[0058]

[0059] Table 2

[0060] like Figure 4 and Figure 5 The diagram illustrates a qualitative comparison of the method of this invention with other methods on Sintel and KITTI in various estimation difficulties. The method of this invention outperforms previous methods in several areas where estimation is challenging. It will be readily understood by those skilled in the art that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0061] Furthermore, the present invention also provides an optical flow estimation device based on hierarchical fully correlated cost volume aggregation, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the optical flow estimation method based on hierarchical fully correlated cost volume aggregation.

Claims

1. An optical flow estimation method based on hierarchical fully correlated cost volume aggregation, characterized in that, The method includes the following steps: (1) Take two adjacent frames in the time dimension and call them the first frame and the second frame, respectively, to calculate the optical flow between the first frame and the second frame; extract the matching features of the two frames and the context features of the first frame; (2) Select the feature map of the target size and construct the fully correlated cost body through matrix multiplication. The fully correlated cost body is spatially divided and reorganized to allow cost aggregation using 2D and 3D convolution. Then, hierarchical aggregation is performed through a local aggregation network based on 2D convolution and a global aggregation network based on 3D convolution. The aggregated cost body is used to construct a cost body pyramid. Step (2) specifically includes: a hierarchical fully correlated cost body aggregation module performs hierarchical aggregation on the fully correlated cost body. After extracting the matching features f1 and f2 of two frames of images, the fully correlated cost body is constructed by pointwise multiplication between each pair of pixels. The cost volume C has dimensions [H1, W1, H2, W2], where dimensions H1 and W1 are the height and width, respectively, and H2 and W2 represent the displacement ranges in the vertical and horizontal directions, respectively. The hierarchical fully correlated cost volume aggregation module performs hierarchical aggregation of the fully correlated cost volume. Specifically, the module divides aggregation into local and global aggregation. First, the fully correlated cost volume is divided into blocks and reorganized to allow aggregation using 2D and 3D convolutions. The fully correlated cost volume C is divided into N P×P blocks in the displacement range dimension [H2, W2]. The fully correlated cost volume is reorganized into a local cost volume C with dimensions [N, P × P, H1, W1]. local The local aggregation submodule applies 2D convolutions to the local cost volume C. local Cost aggregation yields a regularized cost body. Further through 2D convolutional layers The dimensions become [1, N, H1, W1]. The global aggregation submodule performs global aggregation on this cost volume through 3D convolution to obtain a regularized cost volume. Its dimensions are Further through the The cost volume is obtained by performing average pooling along the displacement range dimension. and The dimensions are respectively and Ultimately, a level 4 cost body pyramid was obtained. Provide displacement information of different magnitudes for the optical flow iterative optimization module; (3) Based on the current estimated optical flow, multi-scale matching information is searched from the cost volume pyramid. After encoding with the context features of the first frame image, the optical flow residual is updated using a convolutional GRU-based update model. After multiple iterations, the final optical flow estimation result is obtained. (4) Upsample the optical flow obtained in the last iteration to restore it to the original resolution, output the full-resolution optical flow map to obtain the optical flow estimate from the first frame image to the second frame image, and calculate the L1 loss between the optical flow estimate and the label to supervise the network.

2. The optical flow estimation method based on hierarchical fully correlated cost volume aggregation as described in claim 1, characterized in that, In step (1), the extraction of matching features from the two frames and contextual features from the first frame specifically includes: The feature encoding module includes two sub-modules: a matching feature extraction sub-module and a context feature extraction sub-module. Both have the same structure and are composed of transformer-based networks. For two frames I1 and I2, the matching feature extraction sub-module extracts matching features f1, f2 ∈ R from the two frames respectively. D×H×W Where D is the dimension of the feature, and H and W are the height and width, respectively, which are 1 / 8 times the resolution of the original image; the context feature extraction submodule extracts the context features f from the first frame image I1. c ∈R D ×H×W .

3. The optical flow estimation method based on hierarchical fully correlated cost volume aggregation as described in claim 1, characterized in that, The local aggregation submodule consists of 1×1 2D convolutions and 11×11 large kernel depthwise segregating convolutions, with GELU as the activation function. The global aggregation submodule consists of 4 layers of 3D convolutions.

4. The optical flow estimation method based on hierarchical fully correlated cost volume aggregation as described in claim 1, characterized in that, Step (3) specifically includes: In each iteration, the optical flow iterative optimization module searches for matching information corr from the cost volume pyramid based on the currently estimated optical flow flow, and compares it with the context features f of the first frame image. c The input is fed into the convolutional GRU-based update model, which outputs the hidden state h. The residual Δflow of the current optical flow is obtained from the hidden state h through convolution. The optical flow residual is added to the current optical flow as the initial value of the optical flow for the next iteration. After multiple iterations, the optical flow estimation result is obtained.

5. The optical flow estimation method based on hierarchical fully correlated cost volume aggregation as described in claim 1, characterized in that, The upsampling in step (4) specifically includes: At each iteration, upsampling weights W are generated from the hidden state h through convolution. Based on the upsampling weights, a full-resolution optical flow graph is output by weighted combination of the neighbors of the low-resolution optical flow graph.

6. The optical flow estimation method based on hierarchical fully correlated cost volume aggregation as described in claim 1, characterized in that, In step (4), the L1 loss between the optical flow estimate and the label is calculated to supervise the network, specifically including: For the optical flow sequence {f1,...,f} output after N iterations, N Given an optical flow label f gt Supervision is performed using the L1 loss function, which is defined as follows: Where γ = 0.8, ||*|| represents taking the absolute value of *, and N is the total number of iterations.

7. The optical flow estimation method based on hierarchical fully correlated cost volume aggregation as described in claim 1, characterized in that, In step (3), the optical flow value is set to 0 at the beginning of the iteration.

8. An optical flow estimation device based on hierarchical fully correlated cost volume aggregation, characterized in that: It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the optical flow estimation method based on hierarchical fully correlated cost volume aggregation as described in any one of claims 1-7.

Images