An optical flow estimation method and system based on adaptive streaming

Abstract

The application discloses a kind of optical flow estimation method and system based on adaptive flow propagation, comprising: feature extraction is carried out to adjacent frame image, and four-dimensional correlation body between source frame and target frame is constructed;Based on correlation body and cyclic updating module, obtain initial optical flow;On this basis, introduce adaptive flow propagation module, carry out neighborhood development to current optical flow, and generate structure guide weight according to source frame structure feature, carry out weighted aggregation to neighborhood optical flow to obtain adaptive optimization branch optical flow;Through structure guide filtering, local self-attention calculation is carried out on optical flow feature, structure perception mask is injected into key feature, structure enhancement filtering is carried out to optical flow boundary and occlusion area, and final optical flow result is obtained.The application can alleviate the problem of over-smoothing of optical flow result and mispropagation of occlusion area optical flow caused by fixed weight in existing optical flow estimation algorithm based on flow propagation.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an optical flow estimation method and system based on adaptive flow propagation. Background Technology

[0002] Optical flow, as an important tool for describing the motion vector of pixels over time, is widely used in tasks such as target tracking, motion segmentation, 3D reconstruction, and video understanding. Traditional optical flow estimation methods mainly rely on constructing energy functionals such as brightness uniformity and smoothness terms, and solving the optical flow field through numerical optimization. Representative methods include the classic Horn-Schunck method and methods based on pyramid hierarchical optimization. These methods are effective in scenes with rich textures and small displacements, but they tend to fail in scenes with large displacements, complex occlusions, and non-rigid motion.

[0003] With the development of deep learning, optical flow estimation methods based on convolutional neural networks, such as the FlowNet series and PWC-Net, have improved the accuracy and robustness of optical flow estimation through end-to-end learning. In recent years, RAFT (Recurrent All-Pairs Field Transforms) has proposed an optical flow estimation framework based on four-dimensional correlated volumes and iterative updates, significantly improving the ability to model detailed and large-displacement scenes. Building on this, some works have borrowed the PatchMatch concept, introducing neighborhood flow propagation into optical flow estimation. By propagating the optimal optical flow within candidate neighborhoods, local search and propagation fusion are achieved.

[0004] However, existing optical flow estimation algorithms based on flow propagation generally use fixed weights to aggregate neighborhood optical flows, assuming that each candidate optical flow in the neighborhood contributes equally to the target optical flow or is determined solely by motion similarity. This design presents the following problems in real-world scenarios:

[0005] (1) In large areas of flat or repetitive textures, fixed weights can easily lead to excessive smoothing of the optical flow field, weakening the real motion differences at the boundaries and destroying structural details.

[0006] (2) In areas where there is occlusion, the actual motion of neighboring pixels is often inconsistent. Fixed weights or weights based only on local similarity are prone to introducing erroneous optical flow from occluded objects or backgrounds, resulting in false propagation.

[0007] (3) Some improved methods attempt to construct the motion features required for propagation by continuously deforming the feature map, but the feature map needs to be transformed multiple times during the iteration process, which has a large computational cost and affects the inference efficiency.

[0008] In summary, how to adaptively select and aggregate neighborhood optical flows during flow propagation, while making full use of structural information to suppress error propagation and reducing computational overhead while ensuring accuracy, is an urgent problem to be solved in the field of optical flow estimation. Summary of the Invention

[0009] The purpose of this invention is to address the problems of optical flow oversmoothing and mispropagation in occluded areas caused by fixed weights in existing optical flow estimation algorithms based on flow propagation, and to propose an optical flow estimation method and system based on adaptive flow propagation.

[0010] The objective of this invention is achieved through the following technical solution: an optical flow estimation method based on adaptive flow propagation, the method comprising the following steps:

[0011] The source frame feature map and the target frame feature map are obtained by feature encoding of two adjacent frames of optical flow to be estimated. A four-dimensional correlation volume is constructed based on the source frame feature map and the target frame feature map and the initial optical flow is set.

[0012] The current optical flow is optimized through the first optical flow optimization branch: the current optical flow is expanded into a neighborhood to obtain a neighborhood candidate optical flow; and a structure-aware mask is calculated based on the structural features of the source frame, which is applied to the neighborhood optical flow to modulate the correlation between the query features and the key features to obtain the branch optical flow and fuse it with the original optical flow to obtain the first output branch optical flow. Based on the four-dimensional correlation volume and the first branch output optical flow, the flow propagation motion features are obtained.

[0013] The current optical flow is optimized through a second optical flow optimization branch that runs parallel to the first optical flow optimization branch: based on the structure-aware mask, the correlation between the query features and the key features of the modulated intermediate optical flow field is calculated to obtain the optical flow of the second output branch, and the guided filtering motion features are obtained based on the four-dimensional correlation volume and the output optical flow of the second branch.

[0014] Extract the context features of the source frame and the RAFT features of the current optical flow; after concatenating the RAFT motion features, flow propagation motion features, guided filter motion features and context features into channels, use a gated loop unit and the initial optical flow to concatenate to obtain the residual optical flow as the current optical flow and perform iterative optimization. After the iteration is completed, the final optimized optical flow is obtained.

[0015] Furthermore, the construction of the four-dimensional correlation tensor specifically involves performing a dot product operation on the feature vectors of all spatial locations in the source frame feature map and the target frame feature map to obtain a four-dimensional correlation tensor indexed by the spatial locations of the source frame and the target frame.

[0016] Furthermore, the neighborhood expansion includes: locally expanding the current optical flow field, generating several regular offsets or learnable offsets on the optical flow plane based on the optical flow vector at each pixel position as the center, and adding them to the central optical flow vector to form a corresponding neighborhood candidate optical flow set;

[0017] The rule offset is a preset set of two-dimensional discrete offsets. The learnable offset can be output by the lightweight offset prediction branch based on the source frame context features or the current optical flow features, predicting K two-dimensional offset vectors for each pixel.

[0018] Furthermore, the process of obtaining flow propagation motion feature samples based on the four-dimensional correlator and the output optical flow of the first branch includes: adding the spatial coordinates of the current pixel in the source frame to the corresponding neighborhood candidate optical flow vector to obtain the matching position coordinates in the target frame as the target frame coordinates; and performing indexing or interpolation sampling from the four-dimensional correlator based on the position of the target frame coordinates to obtain the motion feature vectors corresponding to the pixel and its neighborhood candidate optical flows.

[0019] Furthermore, the construction process of the structure-aware mask includes: calculating the semantic similarity score between local features through the local autocorrelation of the source features, using it as the structure guidance weight, and introducing the Sigmoid activation function into the structure guidance weight to further map the value to the (0,1) interval to form a structure-aware mask.

[0020] Furthermore, the correlation calculation between the modulation query features and the key features includes:

[0021] First, optical flow features are mapped into query features, key features, and value features through three different linear mappings. In the local attention calculation, the key features are injected into a structure-aware mask to obtain local attention weights. Finally, the last two dimensions of the local attention weights are normalized using the Softmax function and applied to the value features to obtain high-dimensional optical flow features enhanced by the local self-attention structure. The high-dimensional optical flow features are then mapped back to the original optical flow space using linear mappings, and the final optical flow adjustment is achieved through the residual structure.

[0022] Furthermore, each step of the method is implemented using a network, and also includes supervised training of the endpoint error between the optical flow prediction and the actual optical flow output of each iteration step. The supervision intensity of the prediction error is dynamically adjusted by exponentially increasing weight coefficients to ensure that the iteration results gradually converge to the actual optical flow field.

[0023] According to another aspect of the specification, an optical flow estimation system for implementing the method is also provided, comprising:

[0024] The feature extraction and correlation volume construction module is used to encode the features of two adjacent optical flow images to be estimated to obtain source frame feature maps and target frame feature maps, construct a four-dimensional correlation volume based on the source frame feature maps and target frame feature maps and set the initial optical flow; extract the context features of the source frame and the RAFT features of the current optical flow;

[0025] The adaptive neighborhood flow propagation module performs neighborhood expansion on the current initial optical flow; and calculates a structure-aware mask based on the structural features of the source frame, which is applied to the neighborhood optical flow features to modulate the correlation calculation between query features and key features, and obtains the flow propagation motion features based on the four-dimensional correlation volume.

[0026] The structure-guided filtering module is used to calculate the correlation between the query features and bond features of the modulated intermediate optical flow field based on the structure-aware mask, and to obtain the guided filtering motion features based on the four-dimensional correlation volume.

[0027] The optical flow iteration update module is used to stitch together the RAFT motion features, flow propagation motion features, guided filter motion features and context features through channels, and then stitch them together with the initial optical flow using a gated loop unit to obtain the residual optical flow and iterate to obtain the final optimized optical flow.

[0028] According to another aspect of the specification, an optical flow estimation device based on adaptive flow propagation is also provided, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the aforementioned optical flow estimation method based on adaptive flow propagation.

[0029] According to another aspect of the specification, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the aforementioned optical flow estimation method based on adaptive flow propagation.

[0030] The beneficial effects of this invention are:

[0031] (1) Adaptive flow propagation reduces oversmoothing: By introducing adaptive weights that depend on structural information during flow propagation, this invention enables neighboring optical flows in semantically consistent regions to have higher participation weights, while suppressing neighboring optical flows in textured or occluded regions. This effectively alleviates the problem of oversmoothing of the optical flow field caused by fixed weight aggregation, making the motion boundary clearer.

[0032] (2) More robust optical flow estimation in occluded areas: This invention utilizes the structural features of the source frame to construct a structure-aware soft mask, which suppresses structurally inconsistent regions in both neighborhood propagation and self-attention filtering, significantly reducing the probability of erroneous optical flow propagation in occluded areas and improving the accuracy of optical flow estimation at occlusion boundaries.

[0033] (3) Reuse of related volumes, balancing accuracy and efficiency: In the flow propagation module, the present invention directly performs neighborhood sampling based on four-dimensional related volumes, without the need for multiple deformation operations on the target feature map. This reduces the computational overhead of repeated interpolation sampling during the iteration process from the source, and improves inference speed and resource utilization while maintaining or improving accuracy.

[0034] (4) Structure-guided filtering enhances optical flow structure representation: The structure-guided filtering module introduces structure-aware self-attention in the optical flow feature space, which enhances the high-dimensional optical flow features in the boundary and texture-rich regions, which is beneficial to recover fine-grained structural information. The final optical flow result is superior to the traditional method that only relies on smooth priors in terms of visual perception and quantitative indicators.

[0035] (5) Wide range of applicable scenarios: This invention is based on mainstream optical flow estimation frameworks such as RAFT. It achieves enhancement through modular adaptive flow propagation and structure-guided filtering, which is easy to be compatible with existing network structures and can be widely applied to practical scenarios that require high-precision optical flow estimation, such as autonomous driving, video understanding, and robot vision. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the overall structure of the optical flow estimation method based on adaptive flow propagation provided in an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of the adaptive flow propagation structure provided in an embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram of a structure-guided filter provided in an embodiment of the present invention.

[0039] Figure 4 Comparison of optical flow estimation effects of different methods provided in embodiments of the present invention in the edge region of foreground objects.

[0040] Figure 5 A comparison diagram showing the optical flow estimation effects of different methods provided in the embodiments of the present invention in the occluded region.

[0041] Figure 6 The diagram shows the optical flow estimation effect provided in the embodiment of the present invention.

[0042] Figure 7 This is a schematic diagram of an optical flow estimation device provided in an embodiment of the present invention. Detailed Implementation

[0043] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0044] like Figure 1 As shown, this embodiment provides an optical flow estimation method based on adaptive flow propagation, which includes the following steps:

[0045] Step S1: Feature extraction and correlation body construction.

[0046] Two adjacent input frames are denoted as the source frame and the target frame, respectively. After preprocessing such as normalization, the two frames are input into a convolutional neural network or other feature encoding network with shared weights to obtain the source frame feature map and the target frame feature map. Then, the similarity of the feature vectors corresponding to each spatial position in the source frame feature map and the target frame feature map is calculated to construct a four-dimensional correlation body. Specifically, let the source frame feature map and the target frame feature map be represented as follows: and Where H and W represent the spatial dimensions of the feature map, and d represents the number of feature channels. For a position (x, y) in the source frame and a position (x′, y′) in the target frame, the corresponding correlation value can be calculated by the dot product of their feature vectors, as follows:

[0047]

[0048] This yields a four-dimensional correlation volume indexed by the source frame spatial position (x, y) and the target frame spatial position (x′, y′), which is used to characterize the feature similarity relationship between any source frame position and any target frame position.

[0049] Step S2: Adaptive flow propagation.

[0050] like Figure 1 As shown, an additional optical flow optimization branch is constructed to perform adaptive neighborhood flow propagation on the intermediate optical flow field, outputting the optimized branch optical flow 2 and corresponding features to further enhance the local structural consistency of the optical flow results.

[0051] like Figure 2 As shown, the adaptive neighborhood flow propagation module takes the current optical flow field, the source frame image or feature map, and the four-dimensional correlator as input, and outputs the branch optical flow optimized by neighborhood propagation, which complements the initial optical flow. The specific process includes:

[0052] (1) Neighborhood Unfolding: The current optical flow field is locally unfolded, with the optical flow vector at each pixel position as the center, and the optical flow plane is unfolded with a window radius. The resulting mesh offset tensor Each tensor location stores the optical flow size of the surrounding radius r, thus obtaining a set of candidate optical flows in the neighborhood.

[0053] (2) Adaptive Neighborhood Propagation: First, the similarity of target feature positions is obtained by mapping the source frame feature target position to the target frame feature position through the candidate optical flow deflection in the whole neighborhood. The semantic similarity score between local features is obtained through local autocorrelation calculation, i.e., the structure-guided weight, which imitates the self-attention mechanism. First, after obtaining the features... and characteristics :

[0054]

[0055] in, and This represents a linear mapping between two d-dimensional spaces, i.e. The structural guiding weights can be expressed as:

[0056]

[0057] By introducing a Sigmoid activation function into the structure-guided weights, the values ​​can be further mapped to... The resulting structure-aware soft mask is formed within a certain interval, indicating the intensity at which the neighboring optical flow participates in the flow propagation process. It can be represented as

[0058]

[0059] Then, we apply this structure-aware soft mask to the optical flow (i.e., the similarity score) to give the optical flow a structure-aware representation, and normalize the last two dimensions using the Softmax function to obtain the normalized optical flow. , Right now

[0060]

[0061] This distribution gives Each neighboring candidate optical flow The contribution weights to the branch optical flow update are applied to the neighborhood optical flow, and then aggregated to obtain... The branch optical flow at the location is as follows

[0062]

[0063]

[0064] Finally, we fuse the original optical flow and the branch optical flow using residual connections to form the final output branch optical flow II, and introduce learnable parameters. , Right now

[0065] (11)

[0066] (4) Optical flow iterative optimization: The generated branch optical flow 2 is sampled in the four-dimensional correlator to obtain the flow propagation motion features. This step of four-dimensional correlator sampling is to perform indexing or interpolation sampling in the four-dimensional correlator to obtain the motion feature vector corresponding to the pixel and its neighborhood. Branch optical flow 2 and the flow propagation motion features are respectively concatenated with other branch optical flows and other features and sent to the GRU for subsequent processing, thereby enhancing the contribution of semantically consistent neighborhood optical flow during flow propagation and suppressing the error propagation of texture inconsistencies or occluded areas.

[0067] Step S3: Structure-guided filtering.

[0068] like Figure 1 As shown, an additional optical flow optimization branch is created to perform structure-guided filtering on the intermediate optical flow field, outputting the optimized branch optical flow 1 and corresponding features to further enhance the local structural consistency of the optical flow results.

[0069] like Figure 3 As shown, to further enhance the structural nature of optical flow, a structure-guided filtering module, namely SGFM, is designed for local optical flow features. While the optical flow optimization method of the adaptive flow propagation module is discrete, this SGFM optimization method is local. This paper calculates the semantic similarity score between local features, i.e., the structure-guided weight, through local autocorrelation of source features, mimicking the self-attention mechanism, and injects a structure-aware mask into the key features. Specifically, firstly, a linear mapping... Input optical flow Mapping to high-dimensional features , Right now

[0070]

[0071] Then use three different linear mappings Features They are respectively mapped to query features Key features Sum value characteristics ,Right now

[0072]

[0073] in, and This represents a linear mapping between two d-dimensional spaces, i.e. .

[0074] Injecting structure-aware soft masks into local attention computation , can be represented as

[0075]

[0076] in This represents the local attention weights, implemented similarly to a local self-attention mechanism. Finally, the last two dimensions of the local attention weights are normalized using the Softmax function and applied to the value features. The high-dimensional optical flow features enhanced by the local self-attention structure can be obtained. , Right now

[0077]

[0078] Using linear mapping The high-dimensional optical flow features are mapped back to the original optical flow space, and the final optical flow adjustment is achieved through the residual structure.

[0079]

[0080] The generated branch optical flow 1 is sampled in the four-dimensional correlator to obtain the structure-guided motion feature. Branch optical flow 1 and the structure-guided motion feature are respectively spliced ​​with other branch optical flows and other features and sent to GRU for subsequent processing.

[0081] Step S4: Iterative update of optical flow.

[0082] The structure-guided filtering motion features obtained by sampling the branch optical flow 1 in the four-dimensional correlation volume, along with the aforementioned source frame context features, RAFT original motion features, adaptive flow propagation motion features, branch optical flow 1, branch optical flow 2, and the current optical flow, are fed into the gated recurrent unit, i.e., the optical flow iterative update network. The optical flow iterative update network includes convolutional layers for feature fusion and recurrent update units. In each iteration, the gated recurrent unit retains better spliced ​​features and selectively forgets some features to generate the residual optical flow update result and new spliced ​​features for the next iteration. The residual optical flow and the current optical flow are spliced ​​to obtain the optimized optical flow, which is then fed into the next loop.

[0083] Specifically, under the condition that the initial optical flow is zero or predicted by a coarse-level network, the residual optical flow field is obtained through multi-step iterative updates. In each iteration, the corresponding sampling position in the target frame is first calculated based on the current pixel position and its corresponding optical flow vector. Then, the four-dimensional correlation volume is sampled using the sampling position and its neighborhood offset as an index to obtain local motion features describing the similarity relationship between the current pixel and the candidate position in the target frame.

[0084] Subsequently, the local motion features, source frame context features, and current optical flow are input into the iterative update network. The residual optical flow is generated through feature fusion and a recurrent gating unit. The residual optical flow is then superimposed on the current optical flow, and the process is repeated multiple times to achieve a gradual refinement and update of the optical flow.

[0085] Step S5: Loss function and training strategy.

[0086] During the training phase, the endpoint errors between the optical flow predictions and the actual optical flow output at each iteration step are supervised, and the supervision strength of the prediction error is dynamically adjusted by exponentially increasing weight coefficients. This design significantly increases the requirement for output accuracy in the later stages of network training, thereby ensuring that the iterative results gradually converge to the actual optical flow field. The loss function is defined as follows:

[0087]

[0088] in Total number of iterations These are the weighting coefficients. In the experiment... Set to 0.8, Indicates the first The optical flow prediction results output by the next iteration step The predicted optical flow represents the actual optical flow at the corresponding pixel location, and both consist of horizontal and vertical components. The endpoint error is used to measure the Euclidean distance between the predicted and actual optical flows. Backpropagation is used to jointly optimize the feature extraction network, the adaptive flow propagation module, and the structure-guided filtering module.

[0089] Through the above steps, this embodiment, while maintaining compatibility with RAFT, introduces a structure-aware mechanism in the two stages of flow propagation and optical flow filtering, which effectively improves the structural representation capability of the optical flow field and the estimation accuracy of the occlusion region.

[0090] Implementation Results Explanation:

[0091] like Figure 4 , Figure 5 and Figure 6 As shown, Figure 4 Comparative results of different optical flow estimation methods are presented in scenes containing moving foreground objects and background occlusion. Figure 4 and Figure 5 In the image, (a) represents the input source frame image. Figure 4 and Figure 5 (b) in the table represents the corresponding true optical flow result. Figure 4 and Figure 5 (c) in the figure represents the optical flow estimation results obtained using only the baseline method of fixed neighborhood aggregation. Figure 4 and Figure 5 In the figure, (d) represents the optical flow estimation result obtained by the method of the present invention. Figure 6 A schematic diagram of the reference frame and optical flow estimation results is provided.

[0092] exist Figure 4 and Figure 5The boxes in the diagram mark key locations such as the boundaries of the foreground object and the occluded area. It can be seen that... Figure 4 In the foreground object edge region shown by the box, the optical flow results of the baseline method suffer from inconsistent direction and blurred boundaries, while the method of this invention exhibits more consistent optical flow vector directions and clearer boundary structures in the corresponding region; Figure 5 Near the occlusion area shown in the box, the baseline method is affected by background optical flow interference, resulting in obvious error propagation. However, this invention effectively suppresses the propagation of erroneous optical flow from the occlusion area by introducing structural guiding weights, making the optical flow amplitude of the foreground object closer to the true value.

[0093] The comparison results show that the method of the present invention can better maintain the structural continuity and consistency of the optical flow field at the junction of the foreground and background and near the occlusion area, making the overall optical flow field clearer and more coherent in visual effect.

[0094] Based on the above method embodiments, an optical flow estimation system is also provided, including:

[0095] The feature extraction and correlation volume construction module is used to encode the features of two adjacent optical flow images to be estimated to obtain source frame feature maps and target frame feature maps, construct a four-dimensional correlation volume based on the source frame feature maps and target frame feature maps and set the initial optical flow; extract the context features of the source frame and the RAFT features of the current optical flow;

[0096] The adaptive neighborhood flow propagation module expands the current initial optical flow to obtain neighborhood candidate optical flows. It then uses the neighborhood optical flows to sample the four-dimensional correlator to obtain the motion feature vectors corresponding to the neighborhood candidate optical flows. Based on the source frame structure features and the target corresponding to the position of the neighborhood candidate optical flows, it calculates a structure-aware mask, which is applied to the neighborhood optical flow feature modulation to calculate the correlation between the query features and the key features. Based on the four-dimensional correlator, it obtains the flow propagation motion features.

[0097] The structure-guided filtering module is used to calculate the correlation between key features of the modulated intermediate optical flow field based on the structure-aware mask, and to perform local attention mechanism calculation on the optical flow in combination with value features. Finally, the guided filtering motion features are obtained based on the four-dimensional correlation volume.

[0098] The optical flow iteration update module is used to perform channel splicing of RAFT motion features, flow propagation motion features, guided filter motion features, context features, branch optical flow 1, branch optical flow 2, and current optical flow, and then use a gated loop unit to iteratively obtain the residual optical flow, which is then added to the current optical flow to obtain the final optimized optical flow.

[0099] Corresponding to the aforementioned embodiment of an optical flow estimation method based on adaptive flow propagation, the present invention also provides an embodiment of an optical flow estimation device based on adaptive flow propagation.

[0100] See Figure 7 The present invention provides an optical flow estimation device based on adaptive flow propagation, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements an optical flow estimation method based on adaptive flow propagation as described in the above embodiments.

[0101] The embodiment of the optical flow estimation device based on adaptive flow propagation provided by this invention can be applied to any device with data processing capabilities. This device can be a computer or similar equipment. It acquires video sequences through a camera and outputs optical flow estimation results in real time, providing motion information for downstream tasks. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7 The diagram shown is a hardware structure diagram of any data processing-capable device where the optical flow estimation device based on adaptive flow propagation provided by the present invention is located. (Except for...) Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0102] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0103] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0104] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements an optical flow estimation method based on adaptive flow propagation as described in the above embodiments.

[0105] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0106] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned optical flow estimation method based on adaptive flow propagation.

[0107] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0108] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An optical flow estimation method based on adaptive flow propagation, characterized in that, The method includes the following steps: The source frame feature map and the target frame feature map are obtained by feature encoding of two adjacent frames of optical flow to be estimated. A four-dimensional correlation volume is constructed based on the source frame feature map and the target frame feature map and the initial optical flow is set. The current optical flow is optimized through the first optical flow optimization branch: the current optical flow is expanded into a neighborhood to obtain neighboring candidate optical flows; a structure-aware mask is calculated based on the structural features of the source frame, specifically by generating the structure-aware mask through local autocorrelation calculation and the Sigmoid function; then the structure-aware mask is applied to the similarity scores corresponding to the neighboring candidate optical flows, and the last two dimensions are normalized using the Softmax function to obtain a normalized optical flow. The normalized optical flow provides contribution weights, which are then applied to the neighboring candidate optical flows for aggregation to obtain branch optical flows, which are then fused with the original optical flow to obtain the first output branch optical flow. Based on the four-dimensional correlation volume and the first output branch optical flow, the flow propagation motion characteristics are obtained. The current optical flow is optimized through a second optical flow optimization branch that runs in parallel with the first optical flow optimization branch: First, the current optical flow is mapped into high-dimensional features through a linear mapping, and then the high-dimensional features are mapped into query features, key features and value features respectively through three different linear mappings. Based on the structure-aware mask, the correlation calculation between the query features and key features of the current optical flow field is modulated. Specifically, the key features are injected into the structure-aware mask in the local attention calculation to obtain local attention weights. Finally, the last two dimensions of the local attention weights are normalized using the Softmax function and applied to the value features to obtain high-dimensional optical flow features enhanced by the local self-attention structure. The high-dimensional optical flow features are mapped back to the original optical flow space using a linear mapping, and the final optical flow adjustment is achieved through the residual structure to obtain the second output branch optical flow. Based on the four-dimensional correlator and the second output branch optical flow, the guided filtering motion features are obtained. Extract the context features of the source frame and the RAFT features of the current optical flow; after concatenating the RAFT motion features, flow propagation motion features, guided filter motion features and context features into channels, use a gated loop unit and the initial optical flow to concatenate to obtain the residual optical flow as the current optical flow and perform iterative optimization. After the iteration is completed, the final optimized optical flow is obtained.

2. The optical flow estimation method based on adaptive flow propagation according to claim 1, characterized in that, The construction of the four-dimensional correlation tensor specifically involves performing a dot product operation on the feature vectors of all spatial locations in the source frame feature map and the target frame feature map to obtain a four-dimensional correlation tensor indexed by the spatial locations of the source frame and the target frame.

3. The optical flow estimation method based on adaptive flow propagation according to claim 1, characterized in that, The neighborhood expansion includes: locally expanding the current optical flow field, generating several regular offsets or learnable offsets on the optical flow plane based on the optical flow vector at each pixel position as the center, and adding them to the central optical flow vector to form a corresponding neighborhood candidate optical flow set; The rule offset is a preset set of two-dimensional discrete offsets. The learnable offset can be output by the lightweight offset prediction branch based on the source frame context features or the current optical flow features, predicting K two-dimensional offset vectors for each pixel.

4. The optical flow estimation method based on adaptive flow propagation according to claim 1, characterized in that, The process of obtaining flow propagation motion feature samples based on the four-dimensional correlator and the output optical flow of the first branch includes: adding the spatial coordinates of the current pixel in the source frame to the corresponding neighborhood candidate optical flow vector to obtain the matching position coordinates in the target frame as the target frame coordinates; and performing indexing or interpolation sampling from the four-dimensional correlator based on the position of the target frame coordinates to obtain the motion feature vectors corresponding to the pixel and its neighborhood candidate optical flows.

5. The optical flow estimation method based on adaptive flow propagation according to claim 1, characterized in that, The construction process of the structure-aware mask includes: calculating the semantic similarity score between local features through the local autocorrelation of the source features, using it as the structure guidance weight, and introducing the Sigmoid activation function to the structure guidance weight, which can further map the value to the (0,1) interval to form the structure-aware mask.

6. The optical flow estimation method based on adaptive flow propagation according to claim 1, characterized in that, Each step of the method is implemented using a network. It also includes supervised training of the endpoint error between the optical flow prediction and the actual optical flow output of each iteration step. The supervision intensity of the prediction error is dynamically adjusted by exponentially increasing weight coefficients to ensure that the iteration results gradually converge to the actual optical flow field.

7. An optical flow estimation system for implementing the method of any one of claims 1-6, characterized in that, include: The feature extraction and correlation volume construction module is used to encode the features of two adjacent frames of optical flow images to be estimated to obtain source frame feature maps and target frame feature maps, construct a four-dimensional correlation volume based on the source frame feature maps and target frame feature maps, and set the initial optical flow. Extract the context features of the source frame and the RAFT features of the current optical flow; The adaptive neighborhood flow propagation module performs neighborhood expansion on the current initial optical flow; and calculates a structure-aware mask based on the structural features of the source frame, which is applied to the neighborhood optical flow features to modulate the correlation calculation between query features and key features, and obtains the flow propagation motion features based on the four-dimensional correlation volume. The structure-guided filtering module is used to calculate the correlation between the query features and bond features of the modulated intermediate optical flow field based on the structure-aware mask, and to obtain the guided filtering motion features based on the four-dimensional correlation volume. The optical flow iteration update module is used to stitch together the RAFT motion features, flow propagation motion features, guided filter motion features and context features through channels, and then stitch them together with the initial optical flow using a gated loop unit to obtain the residual optical flow and iterate to obtain the final optimized optical flow.

8. An optical flow estimation device based on adaptive flow propagation, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements an optical flow estimation method based on adaptive flow propagation as described in any one of claims 1-6.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements an optical flow estimation method based on adaptive flow propagation as described in any one of claims 1-6.

Images