A high resolution optical flow estimation method and apparatus

Abstract

The present application relates to the technical field of computer vision optical flow estimation, and provides a high-resolution optical flow estimation method and device. The method comprises: extracting source image features, first target image features and context features for a source image and a target image; performing pooling on the first target image features of the target image to obtain second and third target image features; calculating vertical and horizontal attention features of three different resolutions based on the three different resolution target image features; performing multiple iterations until a final optical flow estimation result is obtained; in each iteration, based on the last optical flow estimation result, indexing the vertical and horizontal attention features of the three different resolutions along the orthogonal direction, associating the source image features with the indexing result to obtain a dynamic orthogonal cost volume; inputting the dynamic orthogonal cost volume, the context features and the last optical flow estimation result into a ConvGRU model to obtain the optical flow estimation result of this time.

Description

Technical Field

[0001] This invention relates to the field of computer vision optical flow estimation technology, and in particular to a high-resolution optical flow estimation method and apparatus. Background Technology

[0002] Optical flow estimation is used to determine the pixel movement that occurs between two consecutive image frames. It is a core problem in computer vision and its applications include action recognition, autonomous driving, and video editing.

[0003] Deep convolutional neural networks (CNNs) have achieved great success in the field of optical flow estimation. FlowNet was the first network based on CNNs for end-to-end optical flow estimation, inspiring a series of subsequent works. FlowNet2 introduced a warping operation, stacking multiple basic models to improve model performance. Methods such as PWC-Net used a coarse-to-fine cost volume pyramid; however, these methods are prone to missing fast-moving small objects at low resolutions. RAFT proposed constructing a 4D cost volume for all pixel pairs and performing cyclic indexing and optimization of the cost value in a convolutional GRU (Gated Recurrent Unit), significantly improving accuracy and generalization performance. However, due to the use of a 4D cost volume (H×W×H×W), memory consumption increases quadratically with increasing input resolution, making it difficult to apply to high-resolution images.

[0004] SCV and Flow1D devise efficient cost volume representations to replace the original 4D cost volume in RAFT for processing high-resolution images. SCV constructs a sparse cost volume by computing the top-k correlations for each pixel. Flow1D constructs two 3D cost volumes, one vertical and one horizontal, to represent spatial correlations. While they improve memory efficiency, they suffer a significant loss in accuracy compared to the original 4D cost volume in RAFT.

[0005] Attention mechanisms have achieved great success in many areas of computer vision. CCNet proposed an effective cross-attention mechanism to aggregate contextual information. GMA uses attention to aggregate global motion features, while GMFlow uses cross-attention to obtain discriminative features for matching. Flow1D uses 1D cross-attention for global feature propagation and then calculates 1D correlations. However, this global propagation force in Flow1D is position-sensitive, leading to the inability to detect true matches under large displacement conditions.

[0006] In summary, existing optical flow estimation methods consume a lot of resources, especially when estimating optical flow for high-resolution images, such as 4K resolution images, where resource consumption will increase further, making optical flow estimation for high-resolution images difficult.

[0007] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0008] The technical problem to be solved by the present invention is that the existing optical flow estimation methods consume a lot of resources, especially when performing optical flow estimation on high-resolution images, such as 4K resolution images, the resource consumption will be further increased.

[0009] The present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides a high-resolution optical flow estimation method, comprising:

[0011] For the source image and the target image, a feature extraction network is used to extract the features of the source image and the first target image, and a context feature extraction network is used to extract the context features from the source image.

[0012] Based on the first target image features of the target image, a second target image feature and a third target image feature are obtained through pooling operations; wherein, the first target image feature, the second target image feature, and the third target image feature are target image features of three different resolutions;

[0013] Based on the target image features of the three different resolutions, the vertical attention features and horizontal attention features corresponding to the three different resolutions are calculated.

[0014] The process is iterated multiple times until the final optical flow estimation result is obtained. In each iteration, based on the previous optical flow estimation result, the three different resolutions of vertical interest features and three different resolutions of horizontal interest features are dynamically indexed along the orthogonal direction. The source image features are associated with the indexing results to obtain the dynamic orthogonal cost volume. The dynamic orthogonal cost volume, context features, and the previous optical flow estimation result are input into the ConvGRU model to obtain the current optical flow estimation result.

[0015] Preferably, the step of extracting source image features and first target image features using a feature extraction network for the source image and target image specifically includes:

[0016] Source image l s and target image l tThe feature extraction network is input into the source image and target image respectively. The source image and target image feature information are extracted through instance normalization and multi-layer convolution. The source image and target image feature information are mapped to 1 / 8 resolution to obtain source image features and first target image features.

[0017] Preferably, the step of extracting contextual features from the source image using a contextual feature extraction network specifically includes:

[0018] The source image is input into a context feature extraction network, which extracts context information through batch normalization and multi-layer convolution. The context information is then mapped to 1 / 8 resolution to obtain the context features.

[0019] Preferably, the process of obtaining second and third target image features based on the first target image features of the target image through pooling operations specifically includes:

[0020] For the first target image feature F t Perform one average pooling operation with a kernel size of 2×2 to obtain the second target feature at a resolution of 1 / 16. Perform two 2×2 average pooling operations on the first target image features to obtain the third target features at 1 / 32 resolution.

[0021] Preferably, obtaining the vertical attention features and horizontal attention features corresponding to the three different resolutions of the target image features specifically includes:

[0022] For the first target image feature F t Perform vertical focus and vertical aggregation to obtain the first vertical focus feature. For the second target image features By performing vertical focus and vertical aggregation, a second vertical focus feature is obtained. For the third target image features By performing vertical focus and vertical aggregation, a third vertical focus feature is obtained. First vertical attention feature Second vertical attention feature and the third vertical attention feature The vertical attention features comprised of the three different resolutions;

[0023] For the first target image feature F t Horizontal focus and horizontal aggregation are performed to obtain the first level of focus features. For the second target image features Horizontal attention and horizontal aggregation are performed to obtain the second level of attention features. For the third target image features Horizontal attention and horizontal aggregation are performed to obtain the third level of attention features. First level of attention features Second level focus features and the third level of attention features The three different resolutions constitute the horizontal attention features.

[0024] Preferably, the first target image feature F t Perform vertical focus and vertical aggregation to obtain the first vertical focus feature. Specifically, it includes:

[0025] For the first target image features F t The first and second preset convolutional layers are applied respectively to generate corresponding feature maps Q and K, where Q, K ∈ R. H×W×D ;

[0026] Expand K along the vertical direction with radius R0 to obtain Based on Ku, the relation matrix is ​​calculated using the first formula. Wherein, the first formula is Normalization factor;

[0027] Normalizing the first dimension of the relation matrix M using Softmax yields the attention mapping A∈R. (2R0+1)×H×W Note the mapping A = softmax(M);

[0028] The first vertical attention feature is calculated using the second formula. Wherein, the second formula is

[0029] Preferably, based on the previous optical flow estimation result, the three different resolutions of vertical interest features and three different resolutions of horizontal interest features are dynamically indexed along orthogonal directions, and the source image features are associated with the indexing results to obtain the dynamic orthogonal cost volume, specifically including:

[0030] Based on the previous optical flow estimation results, the three vertical interest features at different resolutions are dynamically indexed along the orthogonal direction to obtain three vertical indexing results at different resolutions. The source image features are associated with the three vertical indexing results at different resolutions to obtain horizontal cost volumes at different resolutions. The horizontal cost volumes at different resolutions are then connected to obtain a multi-scale horizontal cost volume.

[0031] Based on the previous optical flow estimation results, the horizontal interest features corresponding to three different resolutions are dynamically indexed along the orthogonal direction to obtain three different resolution horizontal indexing results. The source image features are associated with the three different resolution horizontal indexing results to obtain vertical cost volumes of different resolutions. The vertical cost volumes of different resolutions are connected to obtain multi-scale vertical cost volumes.

[0032] The dynamic orthogonal cost volume is obtained by connecting the multi-scale horizontal cost volume and the multi-scale vertical cost volume.

[0033] Preferably, based on the previous optical flow estimation result, the three vertical interest features at different resolutions are dynamically indexed along orthogonal directions to obtain three vertical indexing results at different resolutions. The source image features are then associated with the three vertical indexing results at different resolutions to obtain horizontal cost volumes at different resolutions. The horizontal cost volumes at different resolutions are then concatenated to obtain a multi-scale horizontal cost volume. Specifically, this includes:

[0034] For the source image features F s The target pixel at position (h,w) is determined based on the previous optical flow estimation result f = (f x ,f y Find the target position (h′, w′) = (h+f) on the target image corresponding to the target pixel. y w+f x );

[0035] Based on the target location (h′, w′), the first level cost volume is calculated. Second level cost volume Third level cost volume

[0036] According to the first level cost volume Second level cost volume and third-level cost volume The multi-scale horizontal cost volume was calculated.

[0037] In a second aspect, the present invention provides a high-resolution optical flow estimation device, including an image feature extraction module, a context feature extraction module, a multi-scale interest feature generation module, and an iterative update module;

[0038] The image feature extraction module is used to extract source image features and first target image features using a feature extraction network for the source image and the target image;

[0039] The context feature extraction module is used to extract context features from the source image using a context feature extraction network;

[0040] The multi-scale attention feature generation module is used to obtain second and third target image features based on the first target image features of the target image through pooling operations; wherein the first, second, and third target image features are target image features of three different resolutions; and based on the target image features of the three different resolutions, vertical attention features and horizontal attention features corresponding to the three different resolutions are calculated.

[0041] The iterative update module is used to perform multiple iterations until the final optical flow estimation result is obtained. In each iteration, based on the previous optical flow estimation result, the three vertical attention features and three horizontal attention features of different resolutions are dynamically indexed along the orthogonal direction. The source image features are associated with the indexing results to obtain the dynamic orthogonal cost volume. The dynamic orthogonal cost volume, context features, and the previous optical flow estimation result are input into the ConvGRU model to obtain the current optical flow estimation result.

[0042] Thirdly, the present invention also provides a high-resolution optical flow estimation apparatus for implementing the high-resolution optical flow estimation method described in the first aspect, the apparatus comprising:

[0043] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the high-resolution optical flow estimation method described in the first aspect.

[0044] Fourthly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the high-resolution optical flow estimation method described in the first aspect.

[0045] The dynamic orthogonal cost volume of this invention has low complexity, thus consuming less resources than existing technologies in high-resolution image optical flow estimation. In addition, this embodiment also performs multi-scale analysis, thereby reducing resource consumption while ensuring the accuracy of the analysis, thus realizing a high-precision and low-resource-consumption optical flow estimation method, which is particularly suitable for high-resolution image optical flow estimation. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0047] Figure 1 This is a flowchart illustrating a high-resolution optical flow estimation method provided in an embodiment of the present invention;

[0048] Figure 2 This is a flowchart illustrating another high-resolution optical flow estimation method provided in an embodiment of the present invention;

[0049] Figure 3 This is a flowchart illustrating another high-resolution optical flow estimation method provided in an embodiment of the present invention;

[0050] Figure 4 This is a flowchart illustrating another high-resolution optical flow estimation method provided in an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram comparing the accuracy of a high-resolution optical flow estimation method provided in this embodiment of the invention with that of various methods in the prior art;

[0052] Figure 6 This is a schematic diagram comparing the optical flow prediction results of a high-resolution optical flow estimation method provided in this embodiment of the invention with those of existing methods.

[0053] Figure 7 This is a schematic diagram comparing the optical flow prediction results of a high-resolution optical flow estimation method provided in this embodiment of the invention with those of existing methods.

[0054] Figure 8 This is a schematic diagram comparing another high-resolution optical flow estimation method provided in this embodiment of the invention with existing methods;

[0055] Figure 9 This is a schematic diagram comparing the optical flow estimation results of a high-resolution optical flow estimation method provided in this embodiment of the invention with those of existing methods when applied to high-resolution images;

[0056] Figure 10 This is a schematic diagram of the optical flow estimation results when a high-resolution optical flow estimation method provided in an embodiment of the present invention is applied to a 4K image;

[0057] Figure 11 This is a schematic diagram of the architecture of a high-resolution optical flow estimation device provided in an embodiment of the present invention;

[0058] Figure 12This is a schematic diagram of the architecture of another high-resolution optical flow estimation device provided in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0060] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0061] Example 1:

[0062] Existing optical flow estimation methods consume significant resources, especially when estimating optical flow in high-resolution images, where resource consumption increases further, making optical flow estimation in high-resolution images difficult. To address this issue, Embodiment 1 of this invention provides a high-resolution optical flow estimation method, as follows: Figure 1 As shown, it specifically includes:

[0063] In step 201, for the source image and the target image, a feature extraction network is used to extract the source image features and the first target image features, and a context feature extraction network is used to extract the context features from the source image. The source image features are eight times downsampled features of the source image. Similarly, the first target image features are eight times downsampled features of the target image. The dimensions of the obtained source image features and the first target image features are both H×W×D, where H, W, and D represent the height, width, and feature channel dimension, respectively.

[0064] In step 202, based on the first target image feature of the target image, a second target image feature and a third target image feature are obtained through a pooling operation; wherein the first target image feature, the second target image feature, and the third target image feature are target image features with three different resolutions. In a specific application scenario, the resolution of the first target image feature is 1 / 8 of the target image; the resolution of the second target image feature is 1 / 16 of the target image; and the resolution of the third target image feature is 1 / 32 of the target image.

[0065] In step 203, based on the target image features of the three different resolutions, the vertical attention features and horizontal attention features corresponding to the three different resolutions are calculated.

[0066] In step 204, multiple iterations are performed until the final optical flow estimation result is obtained. In each iteration, based on the previous optical flow estimation result, the three different resolutions of vertical interest features and three different resolutions of horizontal interest features are dynamically indexed along orthogonal directions. The source image features are associated with the indexing results to obtain the dynamic orthogonal cost volume. The dynamic orthogonal cost volume, context features, and the previous optical flow estimation result are input into the ConvGRU model to obtain the current optical flow estimation result. In each iteration, the weights of the ConvGRU model are also iteratively optimized.

[0067] In the first iteration, the position of each pixel in the source image is used as the initial optical flow estimation result. Based on the initial optical flow estimation result, the three different resolutions of vertical interest features and three different resolutions of horizontal interest features are dynamically indexed along the orthogonal direction. The source image features are associated with the indexing results to obtain the dynamic orthogonal cost volume. The dynamic orthogonal cost volume, context features and the previous optical flow estimation result are input into the ConvGRU model to obtain the first optical flow estimation result.

[0068] Comparing the complexity of the method described in this embodiment with that of the prior art RAFT and Flow1D, the dynamic orthogonal cost volume complexity of the method described in this embodiment is H×W×(4R+2), where R (default value is 8) represents the multi-scale search radius. In contrast, the cost volume complexities of RAFT and Flow1D are O(H×W×H×W) and O(H×W×(H+W)), respectively. For high-resolution input, 4R+2 ≤ H+W ≤ H×W, indicating that the peak memory usage of the method described in this embodiment during inference is lower than that of Flow1D and RAFT, meaning that the resource consumption of this embodiment is much smaller than that of RAFT and Flow1D. Furthermore, this embodiment also performs multi-scale analysis, thereby reducing resource consumption while ensuring the accuracy of the analysis, thus achieving a high-precision and low-resource-consumption optical flow estimation method, particularly suitable for high-resolution image optical flow estimation.

[0069] Specifically, the step of extracting source image features and first target image features using a feature extraction network for the source image and target image includes: extracting source image features and first target image features from the source image and target image. s and target image l tThe inputs are fed into a feature extraction network, which extracts corresponding source image and target image features through instance normalization and multi-layer convolution. These features are then mapped to 1 / 8 resolution to obtain source image features and a first target image feature. The feature extraction network architecture includes two residual blocks at 1 / 2 resolution, two at 1 / 4 resolution, and two at 1 / 8 resolution, for a total of six residual blocks. The resolution is halved with each subsequent layer, while the number of channels increases. The feature channel dimension of the feature extraction network is set to 128. In practical applications, the feature extraction network typically extracts source image features and the first target image features by mapping the source image to 1 / 8 resolution. s and target image l t The features obtained are input into two feature extraction networks with shared weights. The final dimensions of both the source image features and the first target image features are H×W×D, where H, W, and D represent the height, width, and feature channel dimension, respectively.

[0070] The process of extracting contextual features from a source image using a contextual feature extraction network specifically includes: inputting the source image into the contextual feature extraction network; the contextual feature extraction network extracts contextual information through batch normalization and multi-layer convolution; and mapping the contextual information to 1 / 8 resolution to obtain the contextual features.

[0071] The context feature extraction network and the feature extraction network have basically the same structure, differing only in the normalization method. The feature extraction network uses instance normalization, inputting the source image into the context feature extraction network to obtain contextual information, while the context feature extraction network uses batch normalization.

[0072] The process of obtaining second and third target image features based on the first target image features through pooling operations specifically includes: processing the first target image features F... t Perform one average pooling operation with a kernel size of 2×2 to obtain the second target feature at a resolution of 1 / 16. Perform two 2×2 average pooling operations on the first target image features to obtain the third target features at 1 / 32 resolution.

[0073] Based on the target image features at the three different resolutions, vertical attention features and horizontal attention features corresponding to the three different resolutions are obtained, such as... Figure 2 As shown, it specifically includes:

[0074] In step 301, the first target image feature F is... tPerform vertical focus and vertical aggregation to obtain the first vertical focus feature. For the second target image features By performing vertical focus and vertical aggregation, a second vertical focus feature is obtained. For the third target image features By performing vertical focus and vertical aggregation, a third vertical focus feature is obtained. First vertical attention feature Second vertical attention feature and the third vertical attention feature The vertical interest features comprise the three different resolutions; in In the diagram, each feature at point P contains the features at point F. t The feature information of a group of points in the same column as P.

[0075] In step 302, the first target image feature F is... t Horizontal focus and horizontal aggregation are performed to obtain the first level of focus features. For the second target image features Horizontal attention and horizontal aggregation are performed to obtain the second level of attention features. For the third target image features Horizontal attention and horizontal aggregation are performed to obtain the third level of attention features. First level of attention features Second level focus features and the third level of attention features The three different resolutions constitute the horizontal interest features; in In the diagram, each feature at point P contains the features at point F. t Feature information of a set of points on the same line as P.

[0076] Specifically, vertical and horizontal attention are applied to the target image features corresponding to three different resolutions (1 / 8, 1 / 16, and 1 / 32). For each target image feature, its vertical and horizontal attention features are generated. In the vertical attention features, each feature point contains feature information of a group of points in the same column as that point in the target image feature. Similarly, in the horizontal attention features, each feature point contains feature information of a group of points in the same row as that point in the target image feature.

[0077] Using the first target image features F t Calculate the vertical attention features For example, the first target image feature F t Perform vertical focus and vertical aggregation to obtain the first vertical focus feature. like Figure 3 As shown, it specifically includes:

[0078] In step 401, the first target image feature F t The first and second preset convolutional layers are applied respectively to generate corresponding feature maps Q and K, where Q, K ∈ R. H×W×D The first and second preset convolutional layers are both convolutional layers with 1×1 filters that are pre-trained by those skilled in the art through network training.

[0079] In step 402, K is expanded along the vertical direction with radius R0 to obtain Based on Ku, the relation matrix is ​​calculated using the first formula. Wherein, the first formula is As a normalization factor, for a given h, w, the calculated M(h,w) is a 1×(2R0+1) matrix.

[0080] In step 403, the first dimension of the relation matrix M is normalized using Softmax to obtain the attention mapping. Note the mapping A = softmax(M); steps 401-403 above can be understood as a process of vertical focus.

[0081] In step 404, the first vertical attention feature is calculated using the second formula. Wherein, the second formula is Step 404 can be understood as the process of vertical aggregation.

[0082] Based on the previous optical flow estimation result, the three different resolutions of vertical interest features and three different resolutions of horizontal interest features are dynamically indexed along orthogonal directions. The source image features are associated with the indexing results to obtain the dynamic orthogonal cost volume, which specifically includes:

[0083] Based on the previous optical flow estimation results, the three vertical interest features at different resolutions are dynamically indexed along orthogonal directions to obtain three vertical indexing results at different resolutions. The source image features are then associated with the three vertical indexing results at different resolutions to obtain horizontal cost volumes at different resolutions. The horizontal cost volumes at different resolutions are then connected to obtain a multi-scale horizontal cost volume.

[0084] Based on the previous optical flow estimation results, the horizontal interest features corresponding to three different resolutions are dynamically indexed along the orthogonal direction to obtain three different resolution horizontal index results. The source image features are associated with the three different resolution horizontal index results respectively to obtain the vertical cost volume of different resolutions. The vertical cost volumes of different resolutions are connected to obtain the multi-scale vertical cost volume.

[0085] The dynamic orthogonal cost volume is obtained by concatenating the multi-scale horizontal cost volume and the multi-scale vertical cost volume. That is, in each iteration, for a pixel on the source image feature, we dynamically find its corresponding target position on the target image based on the current optical flow. With the target position as the center, we perform 1D association between the source image feature and the dynamically indexed multi-scale interest feature along the horizontal and vertical directions respectively to obtain the dynamic orthogonal cost volume.

[0086] Taking the calculation of horizontal cost volume through vertical features as an example, using the vertical interest features calculated at 1 / 8, 1 / 16 and 1 / 32 resolutions, the horizontal cost volume along the horizontal direction at the three resolutions can be calculated respectively. Connecting the three horizontal cost volumes together yields the multi-scale horizontal cost volume. Similar to vertical interest and horizontal features, we perform vertical correlation on the target features of horizontal interest to obtain the multi-scale vertical cost volume. Finally, we connect the multi-scale horizontal cost volume and the multi-scale vertical cost volume to obtain the dynamic orthogonal cost volume.

[0087] Taking the calculation of multi-scale horizontal cost volume as an example, based on the previous optical flow estimation result, the three vertical interest features at different resolutions are dynamically indexed along orthogonal directions to obtain three vertical indexing results at different resolutions. The source image features are then associated with the three vertical indexing results at different resolutions to obtain horizontal cost volumes at different resolutions. Finally, the horizontal cost volumes at different resolutions are concatenated to obtain the multi-scale horizontal cost volume. Figure 4 As shown, it specifically includes:

[0088] In step 501, for the source image feature F s The target pixel at position (h,w) is determined based on the previous optical flow estimation result f = (f x ,f y Find the target position (h′, w′) = (h+f) on the target image corresponding to the target pixel. y w+f x In the first iteration, f = (0,0) is used as the initial optical flow estimation result, and the target position (h′,w′) = (h,w) of the target pixel on the target image is found accordingly, thus performing the first iteration.

[0089] In step 502, the first level cost volume is calculated based on the target position (h′, w′). Second level cost volume Third level cost volume Wherein, r0 is the search radius corresponding to the first vertical interest feature, i.e., R0 mentioned above; r1 is the search radius corresponding to the second vertical interest feature, also referred to as R1 below; and r2 is the search radius corresponding to the third vertical interest feature, also referred to as R3 below.

[0090] In step 503, based on the first level cost volume Second level cost volume and third-level cost volume The multi-scale horizontal cost volume was calculated.

[0091] Steps 502 and 503 can be understood as follows: Indexing the multi-scale target features along the horizontal and vertical directions, centered on the pixel position (h′, w′). For target features at 1 / 8, 1 / 16, and 1 / 32 resolutions, the search radii are R0 = 4, R1 = 2, and R2 = 2, respectively, with a total search radius of R = R0 + R1 + R2. Based on the vertical target features... The horizontal cost volume along the horizontal direction can be calculated.

[0092]

[0093]

[0094]

[0095]

[0096] Similar to vertical attention and horizontal correlation, we define target features of horizontal engagement. Perform vertical correlation to obtain the vertical cost volume C. v ∈R (2R+1)×H×W Finally, C h With C v By connecting the series, the dynamic orthogonal cost volume C is obtained. o ∈R (4R+2)×H×W .

[0097] The dynamic orthogonal cost volume complexity of this embodiment is H×W×(4R+2), where R (default value is 8) represents the multi-scale search radius. In comparison, the cost volume complexities of RAFT and Flow1D are O(H×W×H×W) and O(H×W×(H+W)), respectively. For high-resolution input, 4R+2 << H+W << H×W, indicating that the peak memory usage during inference of the method described in this embodiment is lower than that of Flow1D and RAFT, i.e., the resource consumption is much smaller than that of Flow1D and RAFT methods.

[0098] A performance comparison chart of the method described in this embodiment with other prior art methods is also provided, such as... Figure 5 As shown, Sintel and KITTI are used to measure the performance of this method compared to other methods. In the charts, this method represents the method described in this embodiment. The lower the value, the higher the accuracy of the corresponding method. It can be seen that the accuracy of the method described in this embodiment is higher than most existing methods, such as PWC-Net+, MaskFlowNet, and Flow1D. For example, it exceeds Flow1D by 21.1% on KITTI and by 18.9% on Sintel (Final). This embodiment also provides a visual comparison chart of the method described in this embodiment and the Flow1D method under different test methods, such as... Figure 6 The method described in this implementation is for the Sintel testing method. Figure 6 Subgraph c) and Flow1D method ( Figure 6 A visual comparison of subgraph b) in the image. Figure 7 The method described in this implementation is for the KITTI testing method. Figure 7 Subgraph c) and Flow1D method ( Figure 7 The visual comparison diagram of sub-image b) shows that, compared with Flow1D, the method described in this embodiment performs better in fine structures and large areas without texture. The arrow marks the corresponding position in the target image obtained by optical flow estimation of the target in the source image.

[0099] Furthermore, this embodiment also provides a comparative diagram of the execution flow of the method described in this embodiment and the Flow1D method, as shown below. Figure 8 As shown, the Flow1D method performs a global search across the entire target image when locating the target pixel's corresponding position, resulting in relatively low search performance but high word search accuracy. This method, however, uses multiple iterative local searches, achieving relatively higher search performance and improving search accuracy through multiple iterations. This ensures accurate optical flow estimation with minimal resource consumption. This embodiment also applies the RAFT method, the Flow1D method, and the method described in this embodiment to the same high-resolution image for visual comparison. Figure 9 For the RAFT method ( Figure 9 Subgraph b) Flow1D method ( Figure 9 Subgraph c) and the method described in this embodiment ( Figure 9 Subgraph d) in the diagram is a visual comparison illustration applied to a 1080×1920 resolution image. Figure 10 This is a schematic diagram illustrating the optical flow prediction results of the method described in this embodiment applied to a 4K (2160×3840) resolution image. Figure 9 As can be seen, the visual effect of the method described in this embodiment is comparable to that of the high-precision RAFT method, both capable of reflecting detailed image features. The Flow1D method, however, has relatively lower accuracy. Furthermore, actual testing shows that the memory consumption of the method described in this embodiment is only 1 / 6 that of the RAFT method. When processing 4K images, the RAFT method is unable to handle 4K images due to its enormous resource consumption, while the optical flow estimation result obtained by the method described in this embodiment is as follows: Figure 10 As shown, and based on actual testing, the memory required to process the DAVIS dataset for 4K images is 5.4G, which is relatively low compared to other methods in the prior art.

[0100] Example 2:

[0101] Following the high-resolution optical flow estimation method described in Embodiment 1, this invention further provides a high-resolution optical flow estimation device. This is to elaborate on the system architecture of the corresponding method in Embodiment 1 and to further analyze its design principles in detail. It should be noted that the methods in Embodiment 1 are applicable in this Embodiment 2, and will not be described again in this embodiment.

[0102] This embodiment describes a high-resolution optical flow estimation device, such as... Figure 11 As shown, it includes an image feature extraction module, a context feature extraction module, a multi-scale attention feature generation module, and an iterative update module; the image feature extraction module is used to extract source image features and first target image features using a feature extraction network for the source image and the target image; the context feature extraction module is used to extract context features from the source image using a context feature extraction network.

[0103] The multi-scale attention feature generation module is used to obtain second and third target image features based on the first target image features of the target image through pooling operations; wherein the first, second, and third target image features are target image features of three different resolutions; and based on the target image features of the three different resolutions, vertical attention features and horizontal attention features corresponding to the three different resolutions are calculated.

[0104] The iterative update module is used to perform multiple iterations until the final optical flow estimation result is obtained. In each iteration, based on the previous optical flow estimation result, the three vertical attention features and three horizontal attention features of different resolutions are dynamically indexed along the orthogonal direction. The source image features are associated with the indexing results to obtain the dynamic orthogonal cost volume. The dynamic orthogonal cost volume, context features, and the previous optical flow estimation result are input into the ConvGRU model to obtain the current optical flow estimation result.

[0105] The iterative update module further includes a dynamic orthogonal volume construction submodule and a ConvGRU model update submodule. The dynamic orthogonal volume construction submodule is used to dynamically index the three different resolutions of vertical interest features and three different resolutions of horizontal interest features along the orthogonal direction based on the previous optical flow estimation result, and associate the source image features with the indexing results to obtain the dynamic orthogonal cost volume. The ConvGRU model update submodule is used to obtain the current optical flow estimation result based on the dynamic orthogonal cost volume, context features and the previous optical flow estimation result.

[0106] Example 3:

[0107] like Figure 12 The diagram shown is a schematic representation of the architecture of a high-resolution optical flow estimation device according to an embodiment of the present invention. The high-resolution optical flow estimation device of this embodiment includes one or more processors 21 and a memory 22. Figure 12 Take a processor 21 as an example.

[0108] Processor 21 and memory 22 can be connected via a bus or other means. Figure 12 Taking the example of a connection between China and Israel via a bus.

[0109] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the high-resolution optical flow estimation method in Embodiment 1. The processor 21 executes the high-resolution optical flow estimation method by running the non-volatile software program and instructions stored in the memory 22.

[0110] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0111] The program instructions / modules are stored in the memory 22 and, when executed by one or more processors 21, execute the high-resolution optical flow estimation method in Embodiment 1 above.

[0112] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0113] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A high-resolution optical flow estimation method, characterized in that, include: For the source image and the target image, a feature extraction network is used to extract the features of the source image and the first target image, and a context feature extraction network is used to extract the context features from the source image. Based on the first target image features of the target image, a second target image feature and a third target image feature are obtained through pooling operations; wherein, the first target image feature, the second target image feature, and the third target image feature are target image features of three different resolutions; Based on the target image features of the three different resolutions, the vertical attention features and horizontal attention features corresponding to the three different resolutions are calculated. The process is iterated multiple times until the final optical flow estimation result is obtained. In each iteration, based on the previous optical flow estimation result, the three different resolutions of vertical interest features and three different resolutions of horizontal interest features are dynamically indexed along the orthogonal direction. The source image features are associated with the indexing results to obtain the dynamic orthogonal cost volume. The dynamic orthogonal cost volume, context features, and the previous optical flow estimation result are input into the ConvGRU model to obtain the current optical flow estimation result.

2. The high-resolution optical flow estimation method according to claim 1, characterized in that, The step of extracting features from the source image and the first target image using a feature extraction network specifically includes: Source image l s and target image l t The feature extraction network is input into the source image and target image respectively. The source image and target image feature information are extracted through instance normalization and multi-layer convolution. The source image and target image feature information are mapped to 1 / 8 resolution to obtain source image features and first target image features.

3. The high-resolution optical flow estimation method according to claim 1, characterized in that, The extraction of context features from the source image using the context feature extraction network specifically includes: The source image is input into a context feature extraction network, which extracts context information through batch normalization and multi-layer convolution. The context information is then mapped to 1 / 8 resolution to obtain the context features.

4. The high-resolution optical flow estimation method according to claim 1, characterized in that, The second and third target image features are obtained by pooling based on the first target image features of the target image, specifically including: For the first target image feature F t Perform one average pooling operation with a kernel size of 2×2 to obtain the second target feature at a resolution of 1 / 16. Perform two 2×2 average pooling operations on the first target image features to obtain the third target features at 1 / 32 resolution.

5. The high-resolution optical flow estimation method according to claim 1, characterized in that, The step of obtaining vertical attention features and horizontal attention features corresponding to the three different resolutions of the target image features specifically includes: For the first target image feature F t Perform vertical focus and vertical aggregation to obtain the first vertical focus feature. For the second target image features By performing vertical focus and vertical aggregation, a second vertical focus feature is obtained. For the third target image features By performing vertical focus and vertical aggregation, a third vertical focus feature is obtained. First vertical attention feature Second vertical attention feature and the third vertical attention feature The vertical attention features comprised of the three different resolutions; For the first target image feature F t Horizontal focus and horizontal aggregation are performed to obtain the first level of focus features. For the second target image features Horizontal attention and horizontal aggregation are performed to obtain the second level of attention features. For the third target image features Horizontal attention and horizontal aggregation are performed to obtain the third level of attention features. First level of attention features Second level focus features and the third level of attention features The three different resolutions constitute the horizontal attention features.

6. The high-resolution optical flow estimation method according to claim 5, characterized in that, The first target image feature F t Perform vertical focus and vertical aggregation to obtain the first vertical focus feature. Specifically, it includes: For the first target image features F t The first and second preset convolutional layers are applied respectively to generate corresponding feature maps Q and K, where Q, K ∈ R. H×W×D ; Expand K along the vertical direction with radius R0 to obtain Based on Ku, the relation matrix is ​​calculated using the first formula. Wherein, the first formula is Normalization factor; Normalizing the first dimension of the relation matrix M using Softmax yields the attention mapping A∈R. (2R0+1)×H×W Note the mapping A = softmax(M); The first vertical attention feature is calculated using the second formula. Wherein, the second formula is 7. The high-resolution optical flow estimation method according to claim 1, characterized in that, Based on the previous optical flow estimation result, the three different resolutions of vertical interest features and three different resolutions of horizontal interest features are dynamically indexed along orthogonal directions. The source image features are associated with the indexing results to obtain the dynamic orthogonal cost volume, which specifically includes: Based on the previous optical flow estimation results, the three vertical interest features at different resolutions are dynamically indexed along the orthogonal direction to obtain three vertical indexing results at different resolutions. The source image features are associated with the three vertical indexing results at different resolutions to obtain horizontal cost volumes at different resolutions. The horizontal cost volumes at different resolutions are then connected to obtain a multi-scale horizontal cost volume. Based on the previous optical flow estimation results, the horizontal interest features corresponding to three different resolutions are dynamically indexed along the orthogonal direction to obtain three different resolution horizontal indexing results. The source image features are associated with the three different resolution horizontal indexing results to obtain vertical cost volumes of different resolutions. The vertical cost volumes of different resolutions are connected to obtain multi-scale vertical cost volumes. The dynamic orthogonal cost volume is obtained by connecting the multi-scale horizontal cost volume and the multi-scale vertical cost volume.

8. The high-resolution optical flow estimation method according to claim 7, characterized in that, Based on the previous optical flow estimation result, the three vertical interest features at different resolutions are dynamically indexed along orthogonal directions to obtain three different resolution vertical indexing results. The source image features are then associated with the three different resolution vertical indexing results to obtain horizontal cost volumes at different resolutions. The horizontal cost volumes at different resolutions are then concatenated to obtain a multi-scale horizontal cost volume, specifically including: For the source image features F s The target pixel at position (h,w) is determined based on the previous optical flow estimation result f = (f x ,f y Find the target position (h′, w′) = (h+f) on the target image corresponding to the target pixel. y w+f x ); Based on the target location (h′, w′), the first level cost volume is calculated. Second level cost volume Third level cost volume According to the first level cost volume Second level cost volume and third-level cost volume The multi-scale horizontal cost volume was calculated.

9. A high-resolution optical flow estimation device, characterized in that, It includes an image feature extraction module, a context feature extraction module, a multi-scale interest feature generation module, and an iterative update module; The image feature extraction module is used to extract source image features and first target image features using a feature extraction network for the source image and the target image; The context feature extraction module is used to extract context features from the source image using a context feature extraction network; The multi-scale attention feature generation module is used to obtain second and third target image features based on the first target image features of the target image through pooling operations; wherein the first, second, and third target image features are target image features of three different resolutions; and based on the target image features of the three different resolutions, vertical attention features and horizontal attention features corresponding to the three different resolutions are calculated. The iterative update module is used to perform multiple iterations until the final optical flow estimation result is obtained. In each iteration, based on the previous optical flow estimation result, the three vertical attention features and three horizontal attention features of different resolutions are dynamically indexed along the orthogonal direction. The source image features are associated with the indexing results to obtain the dynamic orthogonal cost volume. The dynamic orthogonal cost volume, context features, and the previous optical flow estimation result are input into the ConvGRU model to obtain the current optical flow estimation result.

10. A high-resolution optical flow estimation device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the high-resolution optical flow estimation method according to any one of claims 1-8.

Images